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LEARNING STRUCTURAL DESCRIPTION'S FROM EXAMPLES* 


Abst rac t 


Tbe research here described centers cm how a machine 
can recognise -concepts and learn concepts to be recognised* 
Explanations are found in computer protracts that build and 
manipulate abstract descriptions of scenes such as those 
Children! construct from toy blocks. One program uses Sample 
scenes CO create models of simple configurations like the 
three-brick arch. Another uses the resulting isocels in 
making identificatlone. Throughout emphasis is given to 
the importance of using good descriptions when exploring, 
hew machines can tooe to perceive and understand the visual 
envir&hment, 


*This report reproduces a thesis of tbe same title submitted to 
the Depot teen c of Electrical ting inhering, Massachusetts Institute 
of Technology, in partial fulfillment of the requirements for the 
degree of Doctor of Philosophy, January 19?!}, 



Tiible yf Canton ts 


At kri lwI a d 5 merit 

p n . n m a 4 + 4- ■ ■■ ■ ■ ^ * * + * * 

■ 

4 + 

4- 

« 

2 

Abstract , . . 

r j- j , B |i g a u ■ ■ ■ i i t 4 a ■ 

■ 


B 

4 

3 




b 

4" 

A 

C ra p t nr 1 

Key Ideas ,***,,,..**. 




' 

t 

1.1 

Scene d estr lot len and Crmoarison * 

4- 

P P 

4 

a, 

7 

1 .2 

Concept feneration arid Learning , 

B 

> ■ 

p 

- 

1 D 

U 

Ident If feat for 



4- 

4 

1 3 

1 > 

Psyc hoi on 1 ea 1 'die liny * . . . + * 

■ 

. . 

B 

■ 

1 5 

Chapter 2 

Ifu i 1 J Inn tie strict ions *. 

■ 

I- B 

fa 

■P 

17 

2.1 

1 lie cletwyrk ,*,,**„**** 

, 

a B 

4 

+ 

1 7 

2 >1 

Ft el ir irar y Proccssiho.. 


g 

- 

. 

2t 

2.3 

The Ale or i thus 

+ 

+ 4 

+ 


2? 

Chapter 3 

b 1 sc ov er inn Groups libfects . * 

4 

4- 4 

+ 

4- 

32 

3.1 

S f'■ qii t! H f. C 5 a. B B a a r 4 P ■ ■ ■ ■ 

4 

4i 4- 

■ 

4 

*?. 

3 .2 

Conrr.on Properties 



<fa 

4 


3 .3 

Cther Finds of fir on pine . * * , . 

a 


4 

4’ 


3*4 

DescriMno a Sr cup; the Typical ’enter * 

4 

* 

an 

Chapter 4 

Similarities ^ u i ^fer sney s . . * 

■ 

- - 

* 

- 

1 01 

k J 

[\fit nor k 'latch 1 nn.* * . . 

a 

4- fi 

■ 

+ 

101 

4*2 

T h€ lUlkclptOn b a a fa fa 4 fa * ■ ■ + 

e 

P i 

* 

. 

103 

4.3 

Cyppar 1 son HSoteS * . * , * . . - - 

+ 

4 4- 

•fa 

. 

1 03 

4*4 

A Catalogue of C-note Types * * . 

- 


H 

fa 

114 

C hanter 5 

Learn i nn and ' lo d. e 1 Hu 1 Id ton * * . 

- 

ii 4- 

4- 

+ 

12E 

5, 1 

Learn ino ,**......*.■■ 





1 ZF 

5* 2 

Descriptions and Models . * * * - 

• 

B 4 

* 

■fa 

1 27 

5,3 

[Kismol es. i-.rar-n flues i and !op~e*anin 

lo s 

4 


1 30 

5*4 

riodel J lv el opr cot . , . . , * ■ ■ 





133 

5*5 

The Eleven tar v 'ridel Lo ild frvq I'perpt inns, 

4- 


1 3 5 

5.6 

Multiple C -notes . . , , , , . . * 



4- 


14 7 

5.7 

Contradictions and Back in n lip . * 

4 

fa p 

♦ 


151 

5.3 

ft her Each Up Possibilities . . . 

4 

+ g- 

4 


3 65 











Chapter 6 

6 . 1 
6.2 

6.3 

6.4 
6, 5 
6+6 
6,7 

6.a 

t.,9 

6.10 

6.11 

Chapter 7 

7.1 

7.2 

7.2 

7.4 
7.6 


C 

7 


7,8 

C ha pt er 6 

8,1 

3 ,2 
a.3 
3.4 

Appendix , 
Ref erenc es 


5 i tl togra phy 


Sane Generated Concepts 

Physical Models and Functions! Model 
The House * , , , 

The Pedestal . , * 

T he Ten t . , , . . 

T he Arc h . , . . . 

The Wodge , H . , 

The Composite Column 
The Arcade , , , , 

The Table , , . ♦ 

The Arch in Depth 
Directions for Improvement 

Identification „ , . . * . , + , . 

'Sate hind and Identification Al terna 
Ex a ct latch , , , , . . . . « + , 

Digression . . , , , , , ■ , , - * 

Elementary Irientlf ics tier , . . , 
Identification in Context . , . . 
Learn fm from Mistakes , . . . . ■ 

The heedle In the Haystack .... 
React inn to Identification , . . , 

Closing Remar ks . , . , , . , , . 


A Systen 

Conclusions , .. 

Background Issues . + . . , 
Suggestions for further Work 


in 4 4 


■# » 

■ h 

■ a 

1 r 


ivo s 


■* J 

In HI 


I 57 

157 
168 
1 62 
1 62 
1 67 
172 
1 79 
183 
I8b 
18(1 
186 

1 9° 

1 95 

2 DC 
204 
213 
22 <1 
223 
237 
239 

2 47 

247 
2 47 
2 d 9 
2 5 0 

25 4 

2 6 * 

2 65 



















1 Key Ideas 

How d p we r ec em i ze exampl cs of various concent?? 

How do we learn to make such reconn It inns? 

How can machines Ho these twines? 

Mow iriTinrtBTi t is careful teaching? 

Iti this naper I describe a system that sheds sonrj lleht 
cn these nuestiens ty demon s tret i no how a machine can be 
taught to see and Team nnv visual ccn cents* It vorKs in the 
rloma in of thr ne -d iron siona 1 str uc tures made of bricks, 

wedges, and other simple objects* 

Sood descriptive methods are of central irgortaoce to 
this work. This is demonstrated repeated \y in py system's 
facilities for scene description, description tempnrisen, 
concept learning, anrt idont Ification. 

It is wy opinion that the framework for learning that 1 
describe suggests a unity between learning from examples, 
learning by imitation, and learning by heinr told* This 
unity lies in the necessary ability to generate and 
maripul&t e good abstract descriptions, 

I alsij arcue the Importance of good training sequences 
prepared by good teachers, I think it is reasonable to 
believe that neither machines nor children can be expected to 
learn nucii without them. 

Enually Important is the notion of the near miss* Ey 
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near miss T mean a sample In a training sequence og 1 to like 
the concept to be learned bu t which differs from that concept 
In only a few significant points at most. These hear misses 
prove to convey essential points much more directly than 
repetitive exposure to ordinary examples, 

1,1 Scene description and Comparison 

Much of the system to he described focuses on the 
problem of analysing scenes consist Inn of the simple objects 
that ore finds in a chfla r s toy box. There are two very 
simple examples of such scenes in figure 1-1. 

From such visual images, the system builds a very coarse 
description, {figure 1*2} Structur frm the scene* s description 
in terms of objects is already a certain commitment, for 
Strife tur inq it, in Other terms is possible, Tn any case, 
analysis proceeds* inserting more detail, (flours 1-3} And 
finally there is the very fine detail about the surfaces, 
lines, vertexes, and their relations. 

Such descr i ot ion s permit one to compare and contrast 
scenes throuoh programs that compare and contrast 
descriptions, Qf course, one hopes that the descriptions 
will be similar or dissimilar to the same decree that the 
scenes they represent seen similar or dissimilar to human 
intuition. Then with a general plan for such manipulations* 
there is further hope that the same machinery can be useful 
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In situations ranging far frn-?n visual ones, oivin a the wort a 
eer ta in genera 1 i ty. 

Certainly the necessary "na tchina programs must bp well 
endowed with ability, for a rich description capability 
requires a matching program that can cope with and perform 
reasonably fn an environment where many matches are possible, 
both, good and bad. 

After two Scenes are described, and correspond in g oarts 
related by the natchinn ororraw, differences in the 
descriptions rust ue found, catecorized, and themselves 
described.. The program that does this must be able to 
examine the descriptions of fioure 1-3 with the help 0* a 
matching program and deduce that the difference between the 
scenes is that there is a su r-pnrterf -by relation in one case, 
while there is an in-front-of relation in the other* Dot the 
faculty must be much more powerful than this simple example 
indicates in order to face more complex pairs of scenes 
exhibiting the entire spectrum between the nearly identical 
and the completely different, 

1.2 Concept Generation and Learn inn 

To build a machine that can analyze line drawings and 
build descriptions relevant to some conoarison procedure is 
useful in itself, Byt this is just a step toward the more 
ambitious goal of creating a program that can learn to 
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recognize structures. I will describe a program that can use 
samples of simple concepts to nenerate models. 

Figure 1 -3 and the next few following It show a sequence 
of samples that enables the machine to learn what an arch Is, 
First it gets the general idea by studying the first sample 
In figure 1-4. Then it learns refinements to Its nrlninal 
conception by comparing its current impression of what an 
arch Is with successive samples* It learns that the supports 
of an arch cannot touch from figure 1-5* It learns ttot it 
does not natter much what the top object is from figure 1-6. 
And then from f inure 1-7 it learns the fact that for one 
object to be supported by the others is a definite 
r e q W frenen t* not just a coincidence carry inn through all of 
the sanplos. 

Such new concepts can in torn help in maklnn other* more 
complex abstractions* Thus the machine uses previous 
learning as an aid toward further learning and further 
analysis of the environment. As yet these procedures are 
clumsy, and the descriptions ureomf orta hi y restricted* hut 
the results are encouranfnq enounh to suggest that those 
methods may lead to increasingly powerful per f ormanc e„ 


1*3 Identification 

Idert if tea tion requires additional programs that use the 
results of comparison programs* There are many problems and 
many alternative methods involved because Identification can 
be done In a variety of contexts. 

In one simple form of Identification, the nachfne 
compares the description of so-me scene to be identified with 
a repertoire of models, or stored concepts, Then at the very 
least there must be some method of evaluating the comparisons 
between the uninoivn and the models so that some natch can be 
defined as best* 

But many sophistications lie beyond this skeletal 
scheme. For one thing, the identification can be sensitive 
to context. In figure 1 ■&, for example, one hidden object Is 
more likely to to a wedge than in the other case, although 
both hidden objects present exactly the sa-me line 
configuration* The Id cot i f Icat ion could be further 
prejudiced if the objective is to locate a particular type n-f 
object. Thus the hidden object In figure 1-9 should be 
tentatively Identified as a possible trapezoidal solid, 
rather than a vedne„ if trapezoidal solids are In demand. 


FIGURE I“8 
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1,4 Psychological Model Inn 

$1mu1 a tion of human Intelligence fs not a primary goal 
of this work. Vet for the most part I have d e ■s Inn ed programs 
that see the world fn terms conforming to human usage and 
taste* These programs produce He sc r lotions that use notions 
such as left-of* on -to p- of * behind, tig, and part-of* 

There are several reasons for this. One is that If a 
machine is to learn from a human teacher, then it is 
reasonable that the machine shoud understand and use the same 
relations that the human dees. Otherwise there would he the 
sort of difference in point of view that prevents 
inexperienced adult teachers frnn interaction smoothly with 
small children. 

Moreover, if the math inn Is to understand its 
environment for any reason, then u nder sta nd ing it fn the sane 
terms humans do helps us humans to understand and improve the 
machine's operation* Little is known about how human 
intelligence works, but It would be foolish to ignore 
conjectures about human methods and abilities if those things 
car* help machines, Much has already been learned from 
programs that use what seem like human methods, "here are 
already programs that prove mathematical theorems* play good 
chess, work analogy problems, under stand restricted forms of 
English* and mere, Vet in contrast, little knowledge about 
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intelligence has come from percentren. work and other 
approaches to intelligence that do not exploit the planning 
and hierarchical organization, that i 5 . t ha rac ter i st ic of human 
t hought * 

Another reason for designing programs that describe 
scenes in human terms Is- that human judgement then serves fls 
g standard* There will be no contentment wl t K mac hines that 
only do as well as humans, Hut until machines become better 
than humans at seeing, doing as well is a reasonable ooal, 
and comparing the performance of the machine with that of the 
human is a convenient wav to measure success* 
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2 Building be script ions 

2*1 The Uetrork 

There are many ways to store facts a Lout a scene. One 
simple format Is the unortlrred list: 

A is on top of G 
Al is a side of ft 
G is in front of C 

Such an arranqenent is desperately inefficient because the 
whole of' nemflry must be searched to oat her all facts about 
soma particular cc-ir.ponent of the seems, It is nature I* 
therefore, to record facts In a ncry structured way to 
facilitate retrieval* 

In this connection, one hears such terms as lists, 
trees, rings, and nets, each of wtilth suggests a fern of 
storage. In selecting one, attention ■Trust be paid to several 
criteria* I have already mentioned the problem of rapid 
access. There may also be a need to use memory space 
efficiently* But in the research nhase, nerhar& it is most 
Important that the storage format be 1m some sense natural 
fclth respect to thE Information to be stored* This means 
that the transformation from a situation to its 
representation should be simple, ret awkward. Simple lists 
suffice for a trip to the grocery store, while tree-1 ike 
charts frequently picture command hierarchies nr trenea 1 cgical 
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h fstnries* 

But nany more complex situations require the net, fl 
good example is the descr 1 pt ion of the words in a natural 
language. Each word fs tie scribed easily Irt terms of 
relationships with other 'words which in turn ere similarly 
described, The result is a dictionary In which each word may 
be thought of as a node which Is related to other nodes 
through the pointers that constitute its clef in it Ion * 

Similarly the network seems to hare the appropriate 
blend of flexibility and elegance needed to deal 
stra fghtforwarl y with scenes. It fs the natural format. 

Like words in a dictionary, each object is naturally thought 
of in terms of relationships to other objects and to 
descriptive concepts Hfco large, rectangular a and standing. 

In fioure 2-1 , for example, one has concepts such as OBJ EC T * 
ABC and OBJECT-DEF, Those are represented d fanrauPSt iCJt H y 
as circles, (figure 2-2) Labelled arrows or pointers define 
the relationships between the concepts, (figure 2-3) Other 
pointers indicate membership in general classes or specify 
particular properties* (figure 2-4) And pointers to circles 
representing the sides extend the depth of the description 
and allow more detail, (figure 2-5} 

Now notice that notions like SUPPORTED-Hr, AftPVE* LEFT- 
OF „ BENEATH, and A-KIND-OF may be used not only as relations, 
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but also as concepts- Consider SU PPGP.T ED -li Y* The statement, 
"The WEDflE it SUPPORT ED-BY the BLOCH, 14 uses SUPPORTED-BY as a 
relation- Gut the statement* "SUPPORTED-B V is the opposite 
Of NOT-SU PPORTED*Ey t " uses SL 1 PP DRIED -GY as a concept 
undergoing ok pi lea t ion. Consequently SUPPORTED-BV is a node 
fn the network, as well as a pointer label, and SUPPORTED -B r 
itself Is defined in terms of relations to other nodes, 

T lour e 2-b shows some of the surround f r» n relations and 
OOncept s. 

5U PPORTEU-BY may therefore apoear In diagrams as a 
circle label or as a pointer labet denendlrtq on Its function. 
A circle pierced by an arrow indicates simultaneous use as a 
relation and at a concept- (figure 2-7} 

Thus, descriptions of relationships can be stored in a 
lUHTiogeneous network along with the descriptions of scenes 
that use those relationships. This rermits bin steps toward 
program general ity„ A program to find negatives need only 
know about the relation NEGATIVE-SATELLITE and have access to 
the general memory not. There is no need for the program 
itself to contain a distended table. This way programs can 
operate in many env 1r oftment s, both anticipated and not 
anticipated, Algorithms desianed to manipulate networks at 
the level of scene description can work as easily with 
descriptions of objects, sides, or even of objects 1 
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FIGURE 2-6 
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FIGURE 2-7 










function & t given t he appropriate network. 

2.2 Preliminary Processing 

Consider now the generation of a scene description* The 
starting point is a lino drawing,, without nersnective 
distortion, and the result is to be a network relating and 
describing the various objects with -pointers such as IN* 

FRONT-OF, ABOVE, $U PP OPTED -BY, A-HIND -OF , ABUTS, and HAS- 
PROPERTY-OF. 

First, drawings of t hrcc-d inen si anal scenes are 
cornmu mica ted to the machine using a program by E* K, P* Horn 
together with a special pen whose position on a companion 
tablet tan be read by the machine directly. Then a rrnoram 
written by H, N, Hahabala [1] classifies and labels the 
vertexes according to the number of converging lines and the 
angles between them. Figure 2-3 displays thu available 
categories* Notice that Mahabala's program finds pairs of T & 
where the crossbars lie between col l inear uprights. These 
are called matched Ts. Such nairs occur frequently when one 
object partially occludes another as In firrure 2-9* 

The program then proceeds to create names for all of the 
regions in the scene. H InoroLr si y "region" as used here simply 
refers to any maximal in which one can move from any 

point to any other point without crossing a line. Including 
the background figure Z-9 has eight regions* Various 
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FIGURE 2"8 














































properties are calculated and! stored for these regions* 
Among these are a list of the vertexes surrounding each 
region and & list of the neighboring regions* 
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These results are then supplied to the elegant program 
named SEE developed by A* Guzman [£]. This program 
conjectures about which regions belong to the same objects. 
For figure ?-9 f the end result of the program Is the 
commentary: 

Body 1 consists of A E? C 
Body 2 consists of 0 E F G I s 

Surprisingly the program contains no explicit mo c* e T s for the 
objects it expects to see* It simply examines the vertexes 
and uses the vertex classifications to determine which of the 
neighboring regions are likely to be part of the same object. 
Arrow 5 , for example* stronaly suogest that the two narrow- 
angle regions belong to the same body* (figure £-10) This 
sort of evidence, together with a moderately sophisticated 
executive* can sort out the r^otons in scenes as complicated 
as that in figure £-11* borrowed- from Guzman's thesis. 

Twelve objects are reported and the regions of each are 
r em em be r ed . 

This* then, is the sort of information ready for further 
processing bv ny descrlption-butiding oroorams. 


2*3 T he A lfior ithms 

The following sections do serf be the ideas be hi nd 
programs that look ter the relations ABOVE* SUPPORT, IN™ 

FRO NT-OF, LEFT* ft |A HT, and ^ARRYS* Generally the So programs 
produce descriptions that are in remarkable ha.rirony with 
those of human observers* Sometimes* however* they mate 
conjectures that most humans disagree with. Co these 
occasions one should renOnter that there is no intention to 
precisely mimic psyc hoi oogf ca I phenomena . The noa 1 is simply 
to produce reasonable descriptions that are easy to wort 
with. Right now it Is Important to design and experiment 
with a capable set of programs and postpone the nuestlon of 
how the programs might be refined to he more completely 
1 if el ike. 

2,3*1 Above and 5upport 

T joints are strgnn ctues that one c&ject partly 
obscures another* but then one may as* if the oh sour inn 
occurs because one object is above the other or because one 
is in front of the other. Even in the simple two brick case 
there seems to he an enormous number of c onf igurat ion s* 

Figure 2-12 shows just a few possibilities* 

But fn spite of this variety* there is a simple 
procedure that often seems to correctly decide the ABOVE 
versus IN-FRONT-OF question. Consider the Tines that form 
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the botton border of t he- obsour Inn objects In figure 2-1 2, 
find i no these lines Is the first job of the program* Next 
the program finds other objects whose regions share those 
lines* In general these other objects are below the 
original » obscuring object* 

This algorithm works on all the simple two-tlpck 
situations depicted in figure 2-12. It even works correctly 

on the much more comp 1 lea ted * many-object scene in 
figure- 2-13, shown with the bottom lines Mghliqhted, 

The difficult part is to find the so-called bottom 
lines, which correspond roughly to one's Intuitive notion of 
bottom border* The process proceeds by first not inn those 
lines that lie between two regions of the object in oyestion, 

I call these interior lines* Next the program examines the 
lower of each interior line's vertexes. This is ignered 
unless it is an arrow„ X or a K. Then information about 
bottom linos is nl ea nod from each of the arrows* Xs t and Ks 
in the followt rnr way: 

1* Tf the vertex 1s an arrow* then the two lines 
farming the largest angle* the barbs* are bottom 
line candidates. See figure 2-14. 

2. If the vortex is an X, then the two non-col 1 in ea r 
lines are bottom line candidates. See figure £"15. 

3. If the vertex Is a K* then the two adjacent lines, 
those fornirtti the smallest clockwise and the 
smallest counter-clockwise entiles with the Interior 
Tine arc bottom line candidates. See figure 2-IP, 
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This is really a rule and two Corollaries,, rather than three 

separate rules* X s and Ks result primarily when arrows 

appear fntogn i to, eamouf Tailed by an alionment of objects as 

illustrated by figure 2-15 anti 2-16, Consequently, the 

corresponding rules amount to locating the arrow-form ing 

parts of the vertex and then acting on that basic arrow* 

One further stem is necessary 'before a line can become 

an approved bottori-1 fne + Ps shown by figure 2-17, some of 

the lines nullifying so far must be eliminated* They fall 

because thoy are too vertical, or more precisely* because 

they are too vertical with respect to the arrow's shaft. The 

effective way to weed out bad lines is as follows; 

Rule: 51 imfnate any bottom line candidate ivhfch is more 

vertical than the shaft of the a^row sufoestino that 
ca nd fda t e, 

Of course the pronran extends rudinentary bottom lines 
through certain vertexes. F inure 2-10 shows the obvious 
situations in which the bottom line property is extended 
through the crossbar of a T or the shafts of a pair of 
in a tched T s . 

T hi 5 whole algorithm is based on an assumption that the 
machine observes the scene from above. If the corftour^t 1 on 
dangles from the ceiling, sinole cbnnfs adapt the program to 
discriminate be t we e n U HP Fft and T N - F R An T - OF , rather than AG ny E 
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FIGURE 2-17 
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and IN-fRCNT-OF. One examines ‘instead the hioher vertexes of 
the interior lines, substitutes the term ton lines for bottom 
lines In the vertex inspection rules, and the result inn lines 
usually separate objects from those above th-er. 

Gy search Inn for both the A GOVE and the BELOW 
relations, the mac hi no may often be able to pi-ie ^ its 
□ wn beleht, Consider f inure 2-19, Firm re P.-20 shows 
the same scene with top and bottom lines h in hi iq ht od and 
with the consequent above and below relations* ftt least 
In this ca&e, the machine can correctly deduce that its 
eye is level with object K because both a chain of above 
relations and a chain c* below relations originate at v , 


2,3 ,1 * 1 Discussion 

This algorithm works effectively because of 
Circumstances all likely but not certain to he true in any 
particular scene. The not hod works best when a scene 
consists of bricks and wedges with one side para-1 lei to the 
table. In many other cases, the net hod works anyway, 
Sometimes by co 1 nc Idence and sometimes by principles not yet 
fully ex pi or ed . 

Unfortunately, in explain at f oh I am frequently 
forced to appeal to intuitive notions about what is 
likely and what is not, I know of r.c way to establish a 
reasonable proba bt Hty metric on the situations \ 
discuss. All that can be said now is that any such 
metric should reflect human disposition toward 
configurations exhibition alinnment and symmetry. 

The first likely c ircuinstanc e or principle is that 

objects tend to support other objects by contact through 

relatively horizontal sides. ■Objects net so supported tend 
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to slip, although not always gs demonstrated by figure 2*2] r 
Imagine now that the ton- object in figure 2-22 were 
completely transparent except for a layer of paint on the 
crucial* relatively horizontal bottom side. Since the scene 
is v 1ew?d from above, this bared bottom side will obscure 
part or sometimes all of the Support turn object. See 
figure 2-23, Consequently, some faces of the supporting 
object generally border on lines resuitinn from the edges of 
the bottom side* 

Of course many ef the bottom side’s ednes vanish when 
the object is restored to opacity* N evert hel ess , the ores 
that remain still tend to form part of the seam between the 
supported and the support inn objects. 

Now most of the vertexes of objects are formed by three 
edges meeting together as fri the tip of a pyramid* This 
forms the so-called trihedral angle. Consequently, when two 
observable ertnes of the bottom side form a concave annle, one 
can expect a third edoe of the object to leave the same 
vertex and form the shaft of a -downward directed arrow, 

SI iuht alteration cnuld permit the program to deal 
with many objects with non-tr i hedra 1 vertices. The 
object in figure 2-24, for example* has two Interior 
lines riergfnq at vertex V, Py treating this as a 
generalized arrow* with multiple shafts, the same 
algorithm can be used to define hottnm lines* 

So far the louic is as followsj If one object oh stores 
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another because It is on top of the other* then the sean 
between the two is likely to form the barbs of one or more 
downward directed arrows, belonging to the top object. 

Rever si no this* one would hops for the statement: The barbs 
of downward directed arrows define seams across which one 
likely finds the support Inn object or objects. Unfortunately 
one often finds non*su onnrt ino objects as well. 

£*3.1.2 Refinement 

Figure 2-2 5 suggests a serious Mntl of over enthusiasm. 
The p filar* brick E* elevates the hot ton lines of brick I' and 
they appear between the viewer's eye and a side of the 
massive baqkgrcund brick* brick C. In the two brick case, a 
brick's bottoms lines border on the side of another brick 
only when the one is in fact supported by the other. Libert 
nor e objects are involved, wrong answers may result because 
the bottom lines can wander into regions of objects that do 
not offer support* 

To handle this problem* T u &e a two part procedure. The 
first part Is simply the above algorithm as described so far, 
which now may be thought of as generating a set of possible 
supports. The second part, described below* criticizes these 
possible supports and filters out some of the bad ones. 

Now one reason brick A in figure 2-2 5 clearly does not 
lie on brick C Is that it is absurd to think that an object 
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cars rest on a vertical side of some other object. Fart 2 of 
the support a Ig or it hm mates sure that Just such. cohjectUreh 
supports are eliminated from the list of candidates. To do 
this* it eliminates a candidate ff the only bottnro-1 ine- 
borderlno side seens vertical. It assumes a side is vertical 
if an edge belonging to it is vertical * 

There are two exceptions to this rule that occur whe r an 
object obscures the ertire top of its surportfne object as 
block P, obscures block 2 and as block B obscures block C In 
figure £-25* First, if two bricks are aliened as ere brick 
A and brick E, forming the familiar X vortex, no rejection 
takes place* Second, if the top brick overlaps the support 
as trick B overlaps brick C, at least one unmatched T appears 
and again no rejection takes place. 

If fcy this time iero or one support candidate remains, 
then part 2 terminates, and the support* i f any* is 
announced* There are some common situation?, however, that 
reouiro part 2 to undertake add It fera I computation, Cpm-are 
figure 2-tl with figure 2-20. The vert leal-side filter 
cannot eliminate the possibility of supra or t from brick C In 
either figure because one of C's bottom-line bnrdcrinn sides 
is clearly not vertical. Vet human observers generally el atm 
the top trick in figure 2*2 7 cannot bo other than singly 
supported, whereas they admit there may well be support from 


the large rear block, C, in flan me 2-?A. 

Since the only difference is in the height of brick E, 
this judgement must be the result of a height conpur islcn. 

Stated simply* the program makes ticinht judgements by 
assuming a fl object is supported by the tallest of the surrwt 
candidates surviving so far. It is simply above the others. 
The height of a stack cannot always be commit ed rioorrusly. 
however. In simple cases it is sufficient to locate a 
vertical line belong ing to the object and measure it. Brick 
C in figure 2*27 is such a block, But the verticals of block 
B disappear into 7 Joints and only minimum heights can be 
calculated from such lines without complicated and unexplored 
object extra nol at ion techniques. Consequently, as the 
algorithm reviews the hninhts and minimum heights presenter! 
to it, it first selects the maximum of these. then any 
candidate whose heinht is known exactly Is rejected that 
height is less than the maximum just calculated. All whose 
exact heights are unknown are allowed to pass. 

Figure 2-29 shows why the support algorithm frequently 
resorts to recursion. If the support for block C is to be 
calculated, the hMoht of blocks D and E must he compared. 
But this In turn requires knowledge of their sum or t 50 that 
total heinhts can be added up from the chain of supports and 
used in this last filtering operation of part 2. 
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This completes discussion of the support algorithm 35 it 
now stands* It is. not hard to delude it deliberately, but It 
n 0 v e r t he less operates with a. reliability six ff id lent for use 
by programs that build upon its results* 

In prow ement s can be wade in many ways* The folio wine 
are a few ideas Mob on my priority list: 1, A planned 
modification involves bsina Ls in the search for bottom 
lines* So far the system 93 , 05 , on the scene in fioure 2-30, 
finding no bottom lines* But if one line of an L, is nearly 
vertical and the other Is nearly horizontal, then the nearly 
horizontal line should be an excellent bottom lino* Cf 
course Ls ere frequently buried just as arrows are. Bur fed 
Ls are found fn certain Ts, X S and forks is shown in 
figure 2-31* 2* The discovery of bottom linos can be fouled 

by introduced small objects that obscure crucial vertexes* 

F inure 2-32 shows how. This could be corrected by a 
procedure that extends lines, perhaps after the object is 
identified, 3 . The filter inn operation could be strengthened 
in its ability to detect vertical sides* So far tt knows a 
side is vertical only if the side has a vertical edne. 

Fiaure 2-33 shows how it can run aorourtd as a result* fiono 
of the sides of brick. C appear vertical and the alnorlthm 
consequently reports br fc k A is on ton of brick C as well as 
on brick B* 
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2 , 3.2 In-front-of 

Once a program can discover the SU PPORTED-BY relation* 
then it can frequently deduce IN-FRONT-OF relations by- 
default * That is* if one of two blocks appears to obscure 
another but is not above it* then the relation IN-FRONT-OF is 
a strong possibility. In pursuing this I again use a two 
part program! the First part proposes possible objects that 
a given object may be in front of; the second part rejects 
the bad ones. While simple and direct, this program also 
succeeds admirably on complex scenes. 

Part 1 tries to find all objects that the object tn 
question obscures. First it gathers up most of the obscured 
objects through search for particular types of T joints on 
the periphery of the object, Sup nose one defines a line to 
be physically associated with a particular object when that 
line in the two dimensional drawl no results fron an edge or 
intersection of planes on the object* rather than from some 
obscuring object. Figure 2-34 illustrates. Then the types 
of Ts sought are just those for which cne can be reasonably 
sure tint the crossbar belongs to the object conjectured to 
be the obscurer. 

Figure 2-3 5 shews two kinds of ouali tying T joints. The 
first kind occurs when the wide anqle region associated with 
the T belongs to or is physically associated with the object 
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from which IN-FROM-OF relations are sought. Both of the 
other regions, the ones border log the shaft of the T* belong 
to a second object* This nearly always Indicates the second 
o bj ect Is obscured. 

The second kind of T Joint Illustrated in figure £-35 
occurs when one shaft-bord er inn region belongs to an object 
while the other belongs to the background. This again 
assures the machine that the crosstar tel on as to the 
potentially obscurina object. 

Figure 2-36 indicates by counter example why rothimr can 
be deduced if the s ha *4-border log regions belong to distinct 
objects. The trouble Is that the crossbar of the T is not a 
real edge of block, C, but rather the crossbar Is composed of 
edges belonging to A and B, 

Still another way to locate appropriate Ts is mare 
global. The Idea Is to use whatever means are available to 
find genuine edges belonging to a body and then to see if any 
Ts lie along such lines. Recall that the selection of bottom 
lines Involves inspection of arrows, !( s and Ks at the bottom 
ends of interior lines. Furthermore the rationale behind the 
support algorithm depends on the likelihood that such bottom 
lines arc- phy sica 1 1 y a s soc la ted with the same object as the 
interior line of the arrow* X, nr K, Consegucntly the machine 
can generate a whole famtlyof peripheral lines likely to be 
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physical edoes by simply examining ttie arrows, Ks, and Ks at 
both ends of the interior lines, rather than just those at 
the bettors ends* Then if ary of these physically associated 
lines end at an L, the other line forming the L If added to 
the list. It is very unusual for one leg of an L to belong 
to an object without the other lea belonging alsc, 

plow if any physically associated line is the crossbar of 
a T, then the parent object obscures some other object or 
object Si Figure 2-37 demonstrates what k ind of edges and Ts 
are found by this method. 

Notice that the Is referenced by figure 2-37 are also 
noticed by the previously discussed local Inspection since 
they exhibit the required c cn f lourat ion of objects about the 
Is lines* Figure 2-3S oemonstrates that both methods 
contribute, however, since only the local method worts Oh Ts 
marked L while only the modified be ttom-l f ne finder helps on 
those marked G* 

All of this yields obscured objects which arc candidates 
for relating to the object studied by the relation IN;-FRONT* 
OF* next* part 2 requests help from the support program and 
then immediately rejects all the candidates for which the 
ABQY [ relation is known to hold, This however is often not 
completely sufficient. In figure 2-3 9 > the machine knows 
brick A obscures brick C by virtue of vertex V* Hut ft fs not 
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directly above C, Clearly* the above check must be expanded 
to the presence qf chains of ABOVE relations In order to 
bring the algorithm into line with human taste* In the 
course of this check* the program for ABOVE may be called 
many times if the ABOVE relations have not yet been 
established* Any candidate that survives this check is 
thought to be behind the object studied* 

The most annoying weakness of this algorithm fs that the 
seam between the obscured and the obscuring object may not 
exhibit the required type of T joints. Figure 2-AC shows how 
this can, happen, 1 suspect that further progress can be made 
in these situations of alignment through close consideration 
of Xs and perhaps Ks. 

£.3*3 An Example 

Figure 2-41 provides a somewhat more complex scene for 
the IN-FRONT-FlF and 5U PP&PTEP-BV finding orograms to try. 

The results are as follows: 
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A su pported - by E C 1n*front-of f G 
B K 

C D £ 

D - E 

£ 

f E 

6 

H ! J 

1 
J 

X H E 

The only bad choice Is the neglect of G as a support of F, 
The reason is that the support criticizing program has a 
built in assumption that a supported object's bottom Is 
level, Therefore it believes E is the unly support for F 
because it Is higher than G, the other possibility. 

£♦3*4 Loft and Right 

Two orocrams exist for deciding If one object is left 
of* right of, or neither with respect to another* The first 
computes in a straightforward, simple way, It simply 
conpares the x coordinates of the vertexes of both objects. 
If there is no overlap, that is if 
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xcor(any vertex of one object} 

< 

* cor (any vertex of other object) 

then the first object is to the left of the other, If there 
Is overlap, then no statement can be made. 

This program, based on the no-overlap criterion, could 
be greatly improved through the u&e of an object extending 
program, The machine Is naive to think that object C in 
figure 2^2 is left of object A. Humans tend to fill in the 
obscured portions of object t to form a complete block. 

But even with an ability to imagine the hidden parts of 
objects, such a program refuses to really agree with human 
Judgements, Consider the spectrum of situations to 
figure 2-43, For the first pair of ohjerts, the relations 
LEFT-QF and RIGHT-DF are clearly appropriate. For the last, 
they are clearly not appropriate. To me, the crossover point 
seems to be between the situations expressed by pairs A and 

Now notice that the center of area of one object Is to 
the left of the left-most point of the other object in those 
cases where LEFT-OF seems to hold. It is not so positioned 
If LEFT-OF does not hold, Such a criterion seems In 
reasonable agreement with intuitive pronouncements for many 



FIGURE 2-42 



















FIGURE 2“43 















































of the cases I have studied* It also yields reasonable 
answers In figure 2-44 where in one case A Is to the left of 
B and In the Other case it is rot* Notice that the relation 
Is not symmetric* however* as the- center of area of the much 
longer brick* brick B, Indicates & is to the right of A in 
both cases. 

Figure 2-45 requires extra attention, No natter what 
the center of mass relations* humans are reluctant to use 
either LEFT-OF or PI£HT*QF if one object extends beyond the 
other in both directions* One must additionally specify a 
rule against this* leaving the fallowing for LEFT-OF: 

Say A is left Of & <—> 

1, The center of area of A is left of the leftmost 
point of B, 

2* The rightmost point of A is 1 #ft of the rightmost 
point of B* 

The rule for R1GHT-CF is of course parallel In form, 

!1any people feel their perception *f the relation 
LEFT-QF differs considerably from either Of the two 
possibilities exhibited here, 1 believe the center-of- 
area method is reasonable for the machine now* but it 
would be interesting to more fully explore the euestion 
of what humans think to see if other formulas are 
better. Intuitive notions of LEFT-DF Vary wildly and 
the program can only fee said to generally reflect my 
personal preferences. Indeed* deciding if one object is 
to the left Of another stimulates far more argument than 
do questions involving relations like IK-FRCNT-OF and 
S-l! PPOftT ED-B V, People have difficulty verbalizing how 
they perceive LEFT-OF and tend to waver in their 
methods, hut Implications are that criteria change 
depending on whether the objects Involved are also 
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related by IN-FRQJfT-DF , DN-TOP-DF, B IGfi EP -T H/Ul, and so 
on * 

Professor Karyln Ft in sky has pointed nut to mo that 
the orientations of objects" are also a strong Influence* 
In figure 2-3&, for example* the cube seems left of the 
arch’s entrance even though all its vertexes are clearly 
right of all the arch's vertexes- In view of this 
observation* ny procedure could prntafcly dp better by 
asking basically the same questions as before* but about 
lines through the left-most* right-most, and center-ef- 
area points in the direction of orientation Instead of 
what amounts to vertical projection of the points to the 
X-axis, Put then there is the problem of finding an 
object’s intrinsic orientation, At the moment I know of 
no general heuristics for this, 

2* 3 * 5 Marrys 

The abuts and al igned - wi t h relations arise frequently, 
perhaps because of some human prerfllection to order, As 
Intuitively used* however, neither of these words corresponds 
to the notion i want the machine to deal with. To avoid 
confusion, I therefore prefer to us* the tern marry which 1 
define as follows: 

Definition: An object marry* another if those object? 

have faces that touch each other and have at least one 
common edge. 

Thus the objects in figure 2-4? are said to narry one 
another* Those In figure 2-48 do not because they have no 
common edge. Similarly those In figure 2-49 do not because 
they have no touching faces. 

The MAURY'S relation is sensed by methods resembling 
those pr ev i ou si y d esori bed „ First the vertexes along the 
border are collected. Then the Ts and Ks are further 
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exam ined 3 

The X s arc single to handle* If exactly two of the 
lines are cc11 in car and If the other two separate two 
Objects* then those objects are very likely to deserve the 
MARRYS- relationship because such a vertex is strongly 
associated with aligned stacks or rows. Moure 2-50 
illustrates, those situations* F Inure 2 -Si shows why the two 
objects nu st be separated by the two non -COl Hn ear linos* 
There three sides belong to the sai'ie object and the KARRY5 
relation does not hold. 

If there arc four objects at the vertex and there are 
two pairs of coll in ear lines, then the likely situation is a 
field of objects with those sharing lines marrying each 
other* See figure ?^5£. 

Ks are strongly correlated with the sort of alignment 
illustrated by figure 2-&3, The rule is simple- If there 
are two objects with sides at a K vertex* then they probably 
marry* 

When three objects meet at a T one of the fell owing 
hoi d s 3 

1* The objects with the stem in between marry each 
other and. both obscure the third object* 

2, The third object is the obscuring object* In 
this Case the two Other Objects may Or may not 
ma rr y. 
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3* Ml three obj ect 5 mar ry. 

4 , Something el se , 

Of these, ny programs check for case 1 only* The 
central program looks for chains of IH-FROilT-OF and SUPPORTS 
relations between the s haft - bard eMrq objects and the large- 
angle object* If such chains are found for both, they likely 
both obscure the largo a nolo side and marry each other* 

Figure 2-54 shows examples* 

Now consider the situation where only two objects meet 
at a T with the wide angle side and one of the other s fetes 
belonging to one object. As figure 2*5 5 suggests, this sort 
of T frequently becomes a K when seen from some other angle* 
Like the K* the machine considers it strong enough evidence 
for a. MARRY5 relation. 

Figure 2*56 illustrates both of the T joint situations 
that confirm marrymsnt* 

■Vote that the machine fs conservative in using this 
MARRVS relation; the relation is not placed in ambiguous 
situations Such as those of figure 2-57* 

^3*6 Sha pe 

Before an object can be Identified, the lines that are 
really edges of that object must be sorted from those that 
are edges of obscuring objects* It would not do to think 
object 8 in figure 2*58 has sides shaped tike those in 
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f tgure Z -5 9* 

Ttie Idea of an edge belonging to a body has been 
discussed. The shape program I use first oathers together 
those lines found to be genuine physically associated edges 
by the program that looks for IN -FRONT -DF relations. To 
these It adds any lines that lie between two regions of a 
body, the Interior lines. Then if these Tines include both 
uprights in a pair of matched Ts, it adds a line joining the 
two Ts. And finally* any line shared v*i t h the background or 
other body known to be below or behind Is certainly Included, 
These rules are sufficient to Identify many of the lines that 
belong to any given object, while refecting many that do not 
he Iong. 

Figure 2 -60 shows how this program sees object 0 of 
figure 2-53. Lines L, M, n f and 0 are interior tines* Line 
Fisa segment between matched Ts with the required kind of 
uprights* Q qualifies by way of the IN-F'TONT-OF algorithm, 
while R „ 5* T, U, and V qualify both by way Of the PI-FRONT- 
OF algorithm and the rule adding lines lying between the 
object and the background. Figure 2-61 shews how the rest of 
the scene in figure 2-53 is dissected by this program. 

Notice that the shapes are reasonably well defined. 
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2.3*7 Size 

Piaget has shown that at a certain age children 
generally associate physical size with greatest dimension 
[3]. They will, for example, adamant 1y ma iota in that a tall 
thin beaker has more water in it than a short fat one even 
though they have seen them filled from other beakers of equal 
size* 

Adults dp not develop as far beyond this as might be 
expected. I do not think we really use the notion of volume 
naturally. Apparent area seems much more closely related to 
adult size judgement* Notice that beaker A in figure 2-62 
appears to have about the sane amount of water in it as does 
beakor B, even though it must contain twice as much* Unless 
a su bj ec t con sc iou si y exercises a formula for volume, he is 
likely to report that object B in figure 2-63 is 
a ppr ox imatel y ton times larger than object A f even if told 
both objects are cubes, T he true factor of twenty-seven 
times seems large when the trouble is taken to calculate it* 

Consequently, the size generating program does not 
trouble with volume, Instead It calculates the area of each 
shape produced by the shape detecting algorithm, Next it 
adds together the areas of all shapes belonging to an object 
to get its total area* Then using these areas it can compare 
two objects in size or consult the following table for a 
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3 Discovering Groups of Objects 
Kfoen a scene has more than a few objects, ft is usually 
useful to deepen the heir arc by of the description by dividing 
the objects Into smaller groups which can he described and 
thought of as individual concepts. Figure 3-1* for example, 
seems to divide naturally into three groups of objects* one 
being three objects tied together by SLJ PP DU TED-GY pointers* 
another being three similar objects on top of a fourth, and 
the third be 100 a set of objects in the arch configuration* 
There are other kinds of grouping humans use, but in thfs 
werfc T primarily explore only the three illustrated by this 
figure 3^1* Grouping by identification tv 11li a known model is 
discussed later in the chapter on identification. This 
chapter deals with grouping on the tasis of pointer chains 
and on the basis of property similarities* 

3, 1 Sequences 

A simple kfnd of group consists of chains of SUPPCRTED- 
Blf or Id -FRONT -Of pointers as in the tower of figure 3-1* 

The first act of the grouping: program fs to find sets of 
objects that are so chained together. All such sets with 
three or more elements qualify as groups* 

In the event the sequence of pointers closes on itself* 
a ring is formed. In figure 3-? there Is such a group 
because each of the three objects rests partly on one of the 
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other The result is a circular chain Of SU PPDRTED-BY 

pointers as shown in figure 3-3 + 

li s fng chains to define groups can become fairly complex 
&s illustrated by the scene in figure 3-4, A chain of 
SU PROFITED-EY pointers splits into two branches in scene one 
at the point where object C Is supported fey two objects* D 
and E. In scene two* two chains of SJ PP 0RTED-&Y pointers join 
at M which, supports both l and U The current version of the 
grouping program terminates chains at junction points without 
further fuss. This seems reasonable for It seems natural to 
think of the scenes in Moure 3-4 as a sot of groups 
consisting of A. - E -C „ G-H-I > and J-K-L. 

Another kind of problem arises when objects, tied 
together Ly a simole chain of relations should not 
constitute a group because of other factors. Figure 3-5 
shows one kind of situation that can occur* for which l 
have only ideas but no programs* In this scene the 
machine perceives a sinale object c&nq lomera te* grouped 
together by virtue of an unbroken chain of SU RPORT Efl Y 
pointers* But most humans see a short tower on top of a 
board on top of another tower* This must be partly 
because of the size differences and partly because of 
the fact that the top group is not directly over the 
other objects* In any case, ft would seem that radical 
change in object properties should be possible grounds 
for breaking a chain* With this* one is into territory 
where irrevocable committments should be avoided, 

Perhaps the best thing would be to have the grouping 
program offer alternative groupings of tricky scenes and 
postpone decision until higher level identification 
programs indicate which arrangement leads to the best 
match of the scene against known models* 
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3*2 Common Properties 

When several objects have the same or very nearly the 
same description* they are Immediately solid candidates *or a 
group, The loos on the table in figure 3-6 are typical* All 
are bricks, all are standing, and all are supports for the 
top board* 

This kind of ma h fpg 1 a 11 on Is slightly dangerous in that 
ny criteria for forming a group and admitting members to it 
are a bit flimsy* So far the rules are based on the 
following demands: 

1* All candidates for gr ou p member ship must be related 
to one or more particular objects in the same way* 

For the table case, all four objects are related to 
the hoard by $U PP OR TED-BY, This restriction appears 
necessary because uniform relationship to a single 
object seems to have strong binding power. The 
standing bricks in figure 3-7 naturally constitute 
two groups, not one. 

2* There must be three or more members In the group, and 
the members of the group must share many of their 
properties. 

Figure 3 -8 outlines the procedure for forming such 
groups* The basic idea is to make a generous guess as to 
what objects to include in a group and then to eliminate 
objects which seem atypical until a fairly homogeneous set 
remains* 

To do this, a program first finds a candidate group by 
locating a set of objects that relate to one particular 
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object fn the same way. Next tomes formation of a conmon- 
reU t ionshi ps-1 1 st through a fisting of all relationships 
exhibited by more than half of the candidates 1m the set. 

Figure 3-9 helps explain this process. Objects A 
through F are immediately perceived to be a possible group 
because they all have a relat ion shi p t SU PP GRTED-EY, with a 
single object* G. The relationships exhibited by the 
ca nd Ida tes ar e: 

A f B* and Cj 

1 SUPPORT ED-BY pointer to G 

2 MAR^YS pointer to G 

3 A-KIND-OF pointer to BP, IC K 

4 HAS-PROPERTY-OF pointer to NED IUK-SIZE 

Ds 

1 SJPPORTED-BY pointer to G 

2 MARRYS pointer to G 

3 A -KINU-OF pointer to BRICK 

4 HAS-PR OP ERT V-QF pointer to SMALL 

E and F: 

1 SU PP0RTE0 - B Y pointer to G 

2 KARP tS pointer to G 

3 A-K1ND -OF do in ter to WEDGE 

4 HAS-PROPERTY-OF pointer to 5^1 ALL 

Three relations appear In the conmon-relat ionships-1 1 st 
because they are found in more than half of the candidates' 
relationships 1 i st s: 
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common-relatlonships-t fst: 

1 SUPPORTED-BY pointer to 6 

2 MARRYS pointer to & 

3 A- KIIND-OF pointer to BRICK 

After tKIs eommon-relstlonsMps-1 let is formed* all 
candidates are next compared with it to see how typical each 
is* The measure is simply the fraction of the total number 
of properties of the candidate and the common-r elat lonsht ps- 
1 1st that are shared* Said in a more formal way* the measure 
f s 


number of properties in intersection 
number of properties in union 


where the union and intersection are of 
the candidate's r elat iorsh fos Hist and 
the common-re lat ion s hips-1 1 st * 


Figure 3-10 represents abstractly a situation in which the 
candidate and the c&fnmon-relat ionsh ip s-l 1 st are quite 
different. The shared properties* represented by the shaded 
area, is but a very small fraction of the total area, both 
shaded and unshaded* Figure 3-11 gives the opposite extreme* 
There is considerable overlap and the value Is near one* the 
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max imum pO-SSi hi a+ 

Using this similarity formula to compare the various 
objects of the figure 3-3 example with the common- 
relation s hlps-1 i st. * one has: 

A versus the comm on-r ala tlo ns hip s^l i st - -> 3/4 = .75 
£ versus the comm on-r elat ion sh ips-ti St - -> 3/4 * ,75 
C versus the cum-non-rela t fonship s-1 i st - -> 3/4 * +75 

0 versus the cDirirmn-r elat ionsblos-O i st --> 3/4 = .75 

E versus the common-i' elat ion ship s-1 1 st Z/5 ® ,30 

F versus the comm on-re la tionship s-l 1 st **> 2/5 * .20 

A* B, C» ana D do net have scores of 1 only because the 
comm on-r ela tion sh Ip s -H st does not yet have a prooerty 
Indicating size. The reason is that there f$ no size common 
to more than half ef the currently possible group members* A^ 
0 * £ * D, E* and F + 

The much lover scores of E and F reflect the additional 
fact that as wedges they are different from the standard 
type. They are framed lately eliminated according to the 
following general rul ej 

Eliminate all candidate objects whose similarity scores 
are less than aC£ of the best score any object attains, 

This insures that the group will have members all wi th a 

nearly equal right to belong, 

Next the process is repeated because those oroperties 


95 


common to the remaining candidates may differ from those 
properties common to the original group enough that one or 
more changes should be made to the c o mno n *r e 1 a 11 on s h 1 p s ■ 1 i s t , 
This repetition continues until the elimination process fails 
to oust a -candidate or fewer than throe candidates remain. 

After the first elimination of objects loaves A, C, 
and 3, there Is a new common-relation ships-11st i 
common-r slat Ion ship s*l 1st; 

1 su PP DRIED-BY pointer to G 

2 i*AR*iY5 pointer to G 

3 A-KIND -OF pointer to BEICIC 

4 HAS-PROPEPTY-OF pointer to H ED IUM - SIZE 

Hot ice that there is now a size property since three of the 
four remaining objects have a pointer to ned font size, The 
new comparison scores are: 

A versus the comnon^relation s hip s-1 t st - -!> AM * 1 

B versus the common-rela tion shlps-1 i st --> - 1 

C versus the common -rela tion Shi ps-list -*> 4/4 ■ 1 

D versus the common*r elation ship s-11st --} 3/5 * .6 

This time D- is rejected because Its uncommon size causes 
a low score, leaving a stable group in which the objects are 
all quite Hite one another. 


3.3 Other Kinds of Grouping 

There obviously cannot be a single universal grnow ping 
procedure because attention must be oa fd not only to the- 
scene involved, but also to the needs Of the V&rfOuS programs 
that may request the grouping activity, I have discussed two 
grouping modes that programs can now do in response to 
various demands* There remain many others to bo explored. 

One of the-so involves looking for things that fit 
together* Children frequently do this at play without 
prompting , and adults do It extensively in solving Jigsaw 
puzil es* 

Another kind of grouping, one oartlcularly sensitive to 
the goals of the request, is grouping or the oasis of some 
specified, property* The idea is to pick out all things 
satisfying some criteria, such as all the big standing 
bricks* The result could be a focusing of attention. 

still another way to group Involves overall properties 
that are not obvious from purely local o bservat ion s* 
Techniques here are again laruely unexplored, but it seems 
that overall shape can sometimes Impose unity on a complete 
hodge-podge. Figure 3-12 illustrates this point* All of 
the objects fit together to form a brick-shaped group* This 
is clearly not inherited from a hy COn sistency in how the 
parts are shaped or how they interact with their immediate 
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neighbor s. 

3,4 Describing a Group; The Topical '1 ember 

The machine needs some means of describing groups, The 
method it uses seems to work* but there is room for 
im pr dv enen t, 

First, the parts of the group are gathered together 
under a node treated specifically to represent the group as a 
conceptual unit. Figure 3*13 illustrates this step for a 
group of three objects, A 3 and C* 

Next cones a concise statement of what men Per ship in the 
group means* This is done through the use of a typical- 
member node. Properties end relations that most of the group 
members share contribute to this node's description. If some 
group were composed of three stand fno bricks arranged in a 
tower, then the result would be tho description shown in 
figure 3-14. The typical member is there described as a kind 
of brick, as standing, and as on top of another member of the 
group, Notice also the F3RM pointer to SEQUENCE which 
indicates the kind of group formed. 
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4 Similarities and Differences 
4,1 Network Matching 

Powerful scene description programs are essential to 
scene comparison and identification* Matching is equally 
Important since the machine must knew which parts ef two 
descriptions correspond before It can compute s in i lar it 1e s 
and differences. Figure 4-1 briefly illustrates, A process 
explores the two descriptive networks and decides which nodes 
pf the two tost correspond in the sense that they have the 
same function in their respective networks. The nodes ip a 
pair that so correspond are said to be linked to each other* 
The job of the matching program is simply to find the linked 
pairs, Node LC and node RC in figure 4-1 both have only A- 
KlfiD-CF pointers to BRICK* Since no other nodes have similar 
descriptions, it is clear that LC and R C should he a linked 
pair* Similarly, LS and RB should ho a linked pair since 
both have A-KIND-Of pointers to HEDGE and both have 
SU PPORT ED-Bf pointers to parts of a pair of nodes already 
kno wn to be 1 in Iced , 

Of course the job of the matching program is not SO easy 
when the two scenes and the resulting two networks are not 
Identical, In this case the process forms linked pairs 
Involvino nudes that may not have identical descriptions, but 
seem most similar nevertheless. More details are described 
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in the append ix* 

<1 .2 The Skeleton 

Once the matching process has examined two networks and 
has established the linked pairs of nodes, then description 
of network similarities proceeds, The result is simply a new 
chunk of network that describes those parts of the compared 
networks that correspond. This chunk is called the skeleton 
because it fs a framework for the rest of the comparison 
description, As figure 4-2 suggests, each linked pair 
contributes a node to the skeleton. Certain pointers connect 
the new nodes together* These occur precisely where the 
compared networks both have the same pointer from one member 
of some linked pair to a member of some other linked pair* 
Kotlce that the skeleton Is basically a copy of the structure 
that the compared networks duplicate* 

4.3 Comparison Rotes 

Complete c ompar i son descriptions consist of the skeleton 
together with a second group of nodes attached to the 
skeleton like grapes on a crape cluster, Each of the nodes 
In this second category Is called a c-note, short for 
comparison note* The most common type of c-note Is the 
intersection c^note which describes the situation In which 
both members of a linked pair point to the sane concept with 
the sane pointer* Suppose, for example* that a pair of 
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torresponding objects from two scones are both wedges, Then 
both concepts exhibit an A-KlND-OF pointer to the concept 
WEDGE, {figure 4-3) In Engl ish one can say: 

1, There is something to be said about a certain 
1 in ked pa ir. 

2 , There is an inter sect ion involved, 

3* The associated pointer is A-KIND -OF. 

4. The intersection occurs at the concept WEDaE, 
Figure 4-4 Shows bow each of these sinole facts translates to 
a network entry, First* a pointer named C-KOTE extends from 
the skeleton concept corresponding to the linked pair to a 
new concept that anchors the intersection description. The 
A-KlKD-OF pointer identifies this concept as a kind of 
intersection* Finally Other pointers identify the pointer, 

A-KHID-OF, and the concept* WEDGE, associated with the 
inter sec tion * 

All of the c-notes look like this intersection paradigm, 
,3,1 Digression: Evans' 1 Program 

Embodying difference descriptions in the same network 
format permits operation on those descriptions with the same 
network programs. Thus two difference descriptions can be 
compared as handily as any other pair of descriptions. 

Those familiar with Tom Evans' vanguard program, ANALOGY 
[4] t can understand why this is a powerful feature, rather 
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than simply a contr ifout Ion toward memory homogeneity* Evans’ 
program worked on two dimensional geometric figures rather 
toad drawings of three d iirert s iorva i configurations. 
Nevertheless his Ideas generalize easily and fit nicely into 
the vocabulary used here. 

Figure 4-5 suggests the standard sort of Intelligence 
test problem involved. The machine must select the scene X 
which best completes the statement: A is to B as C is to X. 
In human terns one must discover how B relates to A and find 
art X that relates to C In the same way* 

Using the terminology of nets and descrIptfens* one 
solutfon process can be formalized in the following way: 

First compare A with B and denote the resulting tomparison*- 
describing network by 

d[A;8] . 

Similarly compare C with the answer figures generating 
descriptions of the form d[C:Xj. The result is a complete 
set of comparisons describing the transforriat fon 5 that carry 
one figure into another. Next one should compare the 
description of the transformat Ion from A to B t d£A:E] t with 
the others to see which is most like it. The bast match is 
associated with the best answer to the problem, if m is a 
metric on comparison networks that measures the difference 
between the compared networks* one can say 
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choose X sue.In that 


M(d[dCA:B]:d[C;X]] ) 

Is minimum 

The metric I u so Is not fancy, It Is the one discussed 
later in chapter 7 that serves to identify some scene with 
sane member of a group of models* It 1 works because that 
problem entirely parallels the problem of Identifying a given 
transformation description with some member of a group. The 
identification program* together with a short executive 
routine, handles the problem of figure 4-5 easily, correctly 
reporting scene three as the best answer. Reasonably enough, 
the machine thinks scene one is the second best answer. 

The machine does as well on the slightly altered problem 
in figure 4-6* reporting four as the best answer* 

Of course if the machine's answers are to be those of 
the problem's formuTator, then the machine's describing, 
comparing* and comparison measuring processes should all give 
results that resemble his. Moreover, a really good analogy 
program should have available alternatives to its basic 
describing, comparing, and comparison measuring processes. 
Then fn the event no single answer is much better than the 
others, the program can try some of Its alternatives as one 
or more of its basic functions must not be operating 
according to what the problem maker intended* Evans' program 
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is Superior to mine in this respect because it can often 
compare two drawings in more than one way. It can visualize 
the change In figure 3-7, for example, as either a reflection 
or any one of several rotations. 

Given my formulation of the analogy problem* it is easy 
to see how certain interesting genera 1 1 za t ions can be made. 
After all, once an X is selected, the network symbolized: by 
d[d[A: E] :d[C :X] ) describes the problem* and as a cte scr 1 nt i on* 
it can be compared with the descriptions of other problems, 

By thus applying the comparison programs for the third time, 
one can deal with the question, Analogy problem alpha is most 
like which other analogy problem? Al ter nat i vely, one can 
apply the analogy solving program to probl On descriptions 
instead of scenes and answer the question* Analogy problem 
alpha is to analogy problem beta as analogy problem gamma is 
to which other analogy problem? This involves four levels of 
comparison, Bg t of course there is no limit, and with time 
and memory mac hln es could happily think about extended 
aria logy problems involving an arbitrary number of comparison 
l ev el s, 

t + 3,2 Another Digression: Newell, Shaw, and Simon ‘ s Program 
A classic piece of work in artificial intelligence is 
that of Newell* Shaw* and Simon on the scheme known as the 
General Problem Solver* always a tn-ev iated GPS [ S] , One form 
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of GPS provides another example of how compar f son s may be 
usefully compared with other compart sons. 

From arii abstract point of view, fiPS involves the notion 
that problems may he thought of in terms of some solution or 
goal, G, together with a current state C, Additonally there 
are operators* 0(1}* that convert classes of states into 
others. One may abrevlate their action by writing 

C(i}:F-m(i] --> F-0UTM1, 

meaning that operator 0(f} tends to convert states of the 
form F-IH(1) into states of the form F-QUT(l), 

GPS notices the difference between the current state C 
and the desired state G and then tries to apply an Operator 
relevant to reducing that difference. This nr educes a new 
current state somewnat closer to the desired state* Applying 
this process iteratively, GPS may eliminate the difference 
between C and B, thereby salving the problem, 

In early versions of ftPS the programmers supplied a 
table giving the relevant operations for all the differences 
between C and G that might be observed. But later on News'll 
described an aporoach [ &] that 1 think may to more 
transparently represented Using the sane notions of second 
Order comparison minimisation that is useful In discussing 
analogy problems* The id ea is that the operators, 0 {i ] * may 
he described by the difference between their Input and output 


f ornns s 


d[F-IN M)iF-OUT(l)], 

Then Newell feels it. is heur 1st leal ly sound to apply the 
op&ratosr whose description is most 1 He the difference 
between the current and desired states, 

d[CE&]* 

One can say more formally! 

choose the operator C(1) such that 

M(d[d[F-IN{ i):F-nuT{ 1 >] J (3 [ C: G 2 ] } 

1 s -m fn town. 

Notice that the selection of an operator fs curiously 
like solving ah analogy problem for which one chooses a pair, 
(X(i) t Y(i)) from a set of offered pairs that best completes 
the statement: A is to B as X(i) is ta Y[i). 

4*4 A Catalogue of C-note Types 
4*4,1 The Supplementary-pointer 

Consider the scenes in figure 4-8 and their descriptions 
f n figure 4-9, Scene L has the pointer SUPPORTED-BY between 
LA a nd LB, but scene R does net have a pointer between the 
objects linked to LA and LB. The note describing this 
situation is called a supplementary-pointer c-note and has 
the form shown in figure 4-10* 

Figure 4-11 suggests a related situation* Here the 
linked concepts L and R differ only in that L has an 
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additional pointer Identifying It as standing. This differs 
from tiie supplementary-pointer case in that STANDING is not 
linked to anything in the other scene. A pointer to the 
concept EXIT signals this situation* (figure 4-12) Exits 
involve concepts generated by the scene description program 
as well as concepts like STANDING that reside In the net 
permanently* If one scene contains more objects than 
another, the concepts left over and not matched end up in 
exit pa c kage s. 

4.4*2 Pointer Modifications 

Suppose the networks In figure 4-13 are compared. 

Notice t he MARRY 5 pointer between LA and LB and the 30€S-NhT' 
MARRY pointer between RA and RB. These could be bandied 
Individually as unrelated supplementary-pointer c-nctes, but 
this would ignore the close relationship between HARRYS and 
DOES-KOT-MARRY. Consequently a different type of c-note is 
used that recognizes the- relationship* It is the negative- 
satellite-pair c-note. With it,, the comparison looks as 
shown in figure 4-14. To find such negu t ive-sat ell ite-pa ir 
e-notes, the comparison programs peruse the descriptions of 
unmatched pointers between linked pairs for evidence of 
relationship* For example, MARRYS Is described In part by a 
NEGATIVE-SATELLITE pointer to DOES-UQT-MASRY. Now of course 
there are other pointers that are also just one step removed 
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from a basic relation, ATI such pointers that are 
modifications of the basic relation arc called satellites 
because they cluster around the basic relation to which they 
are attached by t he pointer MODIFICATION-OF, Uncertainty, 
for example, Is expressed by PROBABLY satellites or MAYBE 
satellites. The MUST satellites and the MUST-NOT satellites 
are others of particular importance in model construction* 
These Inform the model matchfng programs that the presence or 
absence of some pointer is vital if some unidentified network 
Is to be associated with a particular model network 
containing such a pointer* figure 4-15 shows some of the 
sat ell Ites of MAftRYS. 

Each type of satellite is associated with a type of c *■ 
note forming an open ended family. Thus in attention to 
negat iv e- sat ell ft e-pa ir c-notes, there are probably- 
satellite-pair c-notes,. may be-satell ite-pa ir c-notes, must- 
satell 1te-pa Ir c-notes* must-n ot-sate 11 ite-pa ir c-notes and 
so on, 

4,4.3 Concept Modifications 

Frequently the members Of a linked pair both have 
pointers to closely related concepts. For example, if a 
brick in one scene fs linked to a tube in another, the 
situation is as shown in figure 4-16, This Is very much like 
the pointer-satellite idea with A-KIND-QF replacing 
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MODIFICATION-Of . In any case, the description genera tor 
recognizes this and similar situations and again generates a 
group of c -note types. The first of these fs the A-KINO-0F 
chain Illustrated by the above situation. This causes the c- 
note of figure ^-17. 

The a - kind-of-chain c-note also includes situations in 
which one concept is related to another not directly, but 
rather through two or three A “-KIUD-fiF relatlonSi Suppose* 
for example* a cube is linked with an object for which no 
identification can be made. There is still an a-kfnd-of- 
chain c-note because cube is linked to the general concept, 
OBJECT, by a sequence of A-KIND-OF relations, [figure 4-13) 

Another kind o* ponular concept modification. is the a- 
kind-of-merge c-note. These a-k ind-of-ner ge c-notes occur if 
there is no A-KIND-OF chain as described above, but each 
concept has a chain of A-K1ND-0F pointers to some third 
concept. For example* WEDGE and DKJCK are both connected to 
the concept* OBJECT, by A* KIND -OF. (figure 4-19) 
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5*1 Learning 

In this chapter I discuss learning to recognize simple 
block configurations* Although this may seem like a very 
special kind of learning* I think the implications are far- 
re nglnq„ 

It is possible to assuime extreme positions on the 
so bj ec t of learn inn. One person may think learning to do 
things is very c om plica ted * while learning to recognize 
things is comparatively simple* because one merely aquires 
templates or some such* Conversely, another person may think 
1 earn inn to do is simple, but learning to recognize Involves 
deep Gestalt 1st problems of synthesis or other imned fmenta * 
rly opinion is that learning by examples, learning by 
being told* learning by imitation* learning by reinforcement 
and other forms are much like one another. 

In the literature there is frequently an unstated 
assumtion that these various forms are fundamentally 
different. But I think the classical boundaries between the 
various kinds of learning will disappear, once superficially 
different kinds of learning are understood tn terms of 
processes that construct and manioulate descriptions. No 
kind of learning need t>e desperately complicated once the 
descriptive machinery is available, but all constitute 


opaque, intractable processes withcut it. 

Then once the problem of using descriptions is 
thoroughly understood, it will be possible to give meaningful 
thought to a deeper problem, that of learning to use 
d escri ption s. It does not seem simple, but using this point 
of view, it seems less than impcs si bl e. 

The notions of learning and teaching are broad and 
confused, Generally, people think of tnese things as 
occurring together, so that whenever something learns, 
something else teaches. But somehow intuitive notions 
fall when it comes to thinking about machines, 

Computers can now play tolerable and improving chess and 
do marvelous symbolic integrations* Vet while people 
freely use the word “teach* in da scribing what tha 
orogramners do, hardly anyone thinks of the machine as 
1 earn 1ng, 

The reason seems to he that the human programmer 
has supplied so much detail that the machine is more a 
mimic than a thing with learning ability. The machine 
acquires its skill without ever really knowing what is 
going on or how it might improve without the laborious 
services of an inf orna t Ion processing surgeon, The 
unfortunate machine is in the position of school pupils 
who know facts and perhaps memorise simple algorithms,, 
but are never programmed to learn* The programs to be 
discussed here are an effort to show that a machine can 
do better and can learn in a realistic sense, given a 
c ha nc e. 

5,2 Descriptions and Models 

I want to make a clear distinction between a description 
of a particular scene and a model of a concept. A model Is 
like an ordinary description in that It carries information 
about the various parts of a configuration. But a model fs 
more in that It exhibits and Indicates those relations and 


properties that must and must not he in evidence in any 
example of the concept Involved. 

Suppose, for example, the description generating 
programs report the following facts In connection with the 
arc h In f igure 5 -1: 

1. Object A i s a br ic k, 

2, Object A is supported by B and C* 

Now suppose the description containing these facts were 
compared with- the scene in figure 5-2, whore object A is a 
wedge, a n-d with the scene in figure 5-3, where object A Mes 
ofi the table* In hoth cases comparison could ce made and 
differences appropriately noted, but the Identification of 
one or the other of these new scenes as arches would be risky 
indeed because so far the machine knows only what one arch 
looks like without knowing what In that description is 
inpor ta nt l 

Humans, however, have no trouble Identifying the scene 
in figure £-2 as an arch because they know that the exact 
shape of the top object in an arch is unimportant. Dm the 
other hand, no one fails to reject the scene In figure 5-3 
because the support relations of the arch are crucial, 
Consequently, it seems that a description must. Indicate which 
relations are mandatory and, which are Inconsequential before 
that description qualifies as a model. This does not require 
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any descriptive apparatus not already on hand. One need only 
substitute emphat it forms like tfU ST-BE-SUPPORTED-BY for basic 
pointers like $U PPORT ED-BV or, in some cases, add new 
pointers* Discovering where and when to perform these 
operations can be somewhat involved, however, and requires 
the bulk of this chapter for discussion* 

5*3 Examples, Near -m i sse s, ard Non-examples 

Suppose it fs desirable to train a machine to recognize 
the letter A without restriction as to type size or font, 

The designer then has two sets of options: He must decide 
how his machine is to works, and he must decide what to show 
the machine. One Idea is to show the machine vast numbers of 
As and hope that it will benefit from such an experience by 
scraohcw noticing the features which spncar repeatedly* But 
this assumes that the frequently seen properties are 
essential ones, which can bo a bad rule* Moreover, there is 
little possibility for skillful teaching, There is no 
obvious way the teacher could ouickly convey a particular 
idea such as the notion that the crossbar of an A is 
Important, even If the teacher realized it* Finally, such 
samples generally only suggest properties a candidate for 
match should have -- it is hard for them to indicate 
forbidden properties. 

This is true because expert description programs 
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would be needlessly fiver burdened ^nd would promote 
confusion If they were always to Indicate all negative 
properties and those not observed. Therefore a 
property’s consistent absence can be easily overlooked* 
If the description program; does list all properties in 
spite of 1 nef f ic iency, then many properties are 
statistically likely to be missing from a short training 
sequence. But the stat i st ic-gatherinq machinery would 
think Such properties should not be present In examples 
of the model, even though their absence was by 
c oinc id enc e. 

One might attempt to train basically the same machine to 
handle a whole repertoire of concepts by showing examples of 
each member* Thus If the repertoire were the alphabet, 
examples of all the letters would be shown, rather than just 
A 5 * This shifts the question to lihich description does an 
unknown best match? 1 It still avoids the more fu ndanenta 1 
question. What is It about each character that Is essential 
and uermlts it to be recognized? The machine now may use As 
and non-As, but the difficulties are only obscured, not 
c Ircumw ented* There remains no way to directly convey an 
Idea, and there remains the fallacy that frequent aopearance 
means Importance. The problem of Indicating what properties 
preclude identification With a particular model fs only 
tangentially and occasionally addressed In that sometimes a 
property converts one description into another, as In the 
case of adding a forbidden mldline crossbar to a C* The 
machine does not know a C. cannot have a crossbar; it only 
knows such a crossbar mates a figure nare like an E. 


In my judgement , near misses are the really important 
examples In learning. In conveying the Idea of an arch* an 
arch certainly should be shown first, But then there should 
be some samples that are not arches, tut -do not miss fceihn 
arc lies by much. Small differences permit the machine to 
localize some part of its current opinion about a concept for 
improvement. If one wants the machine to learn that the 
uprights of an arch cannot marry, one should show it a scene 
that fails to be an arch only in this respect. If the 
machine is to know a C cannot have a crossbar, it should see 
a character that fails to be a C only because of a crossbar* 
Such carefully selectee, near misses can suggest to the 
machine the important Qualities of a concept, can indicate 
what properties are never found, and permit the teacher to 
convey particular ideas quite directly. 

It Is curious how little there is in the 1 iterature 
of machine learning about mechanisms that depend on good 
training sequences. This may be partly because previous 
schemes have been too Inad equate to bear or oven invite 
extensive exploration of this Centrally important topic. 
Perhaps there is also a feeling that creating a training 
sequence is too much tike direct or ogramm In g of the 
machine to Involve real learninq, This is probably an 
exaggerated fear, I agree with those who believe that 
the learning of children is better described by theories 
using the notions of programming and se 1 f'-programming, 
rather than by theories advocating the idea of Self^ 
organization. It is doubtful, for example* that a Child 
could develop much intelligence without the programming 
implicit in his instruction, guidance, closely 
supervised activity, and general interaction with other 
humans. 
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5*4 Model Development 

The machine 4 i model building! program starts with a 
description of some example of the concept to be learned* 

This description is itself the first model of the concept. 
Subsequent samples are either examples of the concept or near 
misses, These examples and near misses repeal weaknesses and 
lead to a new node Is* Section 5,5 shows In some detail how 
the comparison between the current model and the description 
of a new sample produce a new model in the case where only 
one difference Is found* 

One then has a sepyenco of more and more sophisticated 
models, See figure 5-4, Frequently, several responses may 
appropriately address the comparison between the current 
model and: a new sample, When this happens, branches occur in 
the model development sequence and it is convenient to talk 
about a tree of models* Figure 5-5 shows such a tree, 

Section 5,6 discusses how the alternative branches come 
about. The machine selects one branch at each point for 
further development. The meandering path leading from the 
top of the tree down to the current model is called the main 
line. The main lino changes course when a particular 
sequence of branch selections leads to untenable situations. 
Section 5,7 describes how and when this happens. 
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5.5 The El ementar y Hod el Building Operations 

This section cwisldert the case in which the matching 
program finds only one difference between the current model 
and a new example or near miss. The table at the end of this 
section summarizes the results. 

5*5,1 The A-Mnd-of-merge! Current i^odel and Example 

First consider the model and example in figure 5-6* 
Figure 5-7 shows the resulting comparison description* Only 
one difference is found: the object of the model points to 
BRICK while the object of the example points to WEDGE. But 
since both BRICK and WEDGE relate by A-KIRD-OF to OBJECT, the 
a*kind-of-merge c-note occurs, Several explanations and 
companion responses are possible. One is that the source of 
the c-note may in general point to either of the things 
pointed to by the A-KIND-OF pointer In the two scenes. Thus 
the object could be either a V, 1 EDGE or a 1C K. Another 
possibility Is that the A-KINE-QF pointers from the object do 
not matter at all and can be dropped from the model* Still 
another option and the one preferred by the program is that 
the object may be any member of some class in which both 
WEDGE and BRICK are represented. In the example one such 
class is 5imply the concept OBJECT and has already been 
located as the intersection of A-KIND-OF paths. The program 
responds by replacing the pointer in the comparison network 
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that points to the a - V In d-of-merge c-note by an ft-KIKD-CF 
pointer to the Intersection or merge concent* In this ca sc 
an A*KIND*0F pointer is installed between the e-note or in in 
and the concept OBJECT. Then the altered conoariscn network 
is the new node! shown In figure 

The primary response I have selected for the 
machine represents a moderate stand with respect to a 
rather serious Induction problem* I could have been 
more conservative and used an A-KIND-0F pointer to a 
concept representing the ordinary 0^ of BRICK and HEDGE, 
On the othor hand* I could have been far more radical by 
pointing the A *. K IND-OF pointer to THING » the universal 
class, -ly actual choice of pointing to t h* node 
indicated by the merge concept to be the intersection of 
A*KIND-OF chains seems more flexible than either of the 
extreems. 

Since the merge concept is itself defined in terms 
of the network's content* the general izations made will 
change as the net grows* But since the appropriate 
response should after all depend on the universe the 
machine is operating in* the generalization changes are 
likely to be improvene nts* And in any case this 
commitment is only one among many* 

5*5.2 The Su pp 1 emontary-ooint er C-note 

Now suppose scene 1 in figure 5-& represents the current 
model while scene 2 contributes as a near miss. The matching 
routine soon discovers that scene 1 produces a £U FRONTED -8Y 
relation between the two objects whereas scene 2 does not, A 
$upp 1 emontary-point er c-note results* Of course the 
implication is that the concept studied reou ires the two 
objects to stand together under the support relation. 
Consequently* when such a su pp lementary-oolnter c-note turns 
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up* It transforms to the emphatic MUST vor s f on of the pointer- 
involved, Thu 5 the new model 1? the one in figure S-1 0 , 

Of c&urse the su ppl ement-ary pointer can turn up In the 
near miss as well as In the current model- Suppose scene 1 
in figure 5-9 is the near miss instead of the current model. 
One concludes A cannot be on B, The su ppl ememtary-pointer c- 
note now indicates a relation that apparently cannot hold- 
Appropriately, the riUST-NOT version of the su pn 1 ementary 
pointer Is substituted in and the new net appears as in 
figure 5 - 11 

5,|5,3 The Mu st-sa tell ite-pa ir C-note 

Frequently comparison between the current model and a 
new sample displays c-notes that do not reveal any new 
feature, but rather result because of previous refinements in 
the model* 5uppose, for example, that the current model has 
a M-U 5T -MARH Y pointer in a given location, while the sample 
ha s a MARRY S pointer. Now clearly the HARR VS pointer is 
appropriate in the description and the mu st-sat ell Ite-pa ir c - 
note consequent to matching it with MU ST -MARRY should be 
replaced again by HU5T-MARRY, Thus the emphatic form in a 
mu st-satell ite-pa ir situation is retained and not interfered 
with by refinement operations attempted subsequent to its 
forma t ion. 
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5,5,4 The A-kind-of-merge: Current ^odel and Near ^ I s 5 

Sometimes a c-note offers two or more nearly equal 
explanations. Consider the super simple current model and 
rear miss in figure 5-12, The c-ncte is an a-kind-of-merge 
announcing that the current podel points with HAS-' 515 r.Tf r, TY-G r 
to STANDING, the near miss to LYING* and both LYING and 
STANDING have A-KIND -OF paths to ORIENTATIONS. How the near 
ntss may fail either because it Is lying or because it is not 
standing* Responding to these explanations* the model 
builder night replace the a-k ind-of -merge c-note by a MU5T- 
NOT-HAV E-PP.GPERTY-GF pointer to LYING or by a MUST-HAVE- 
PU0FERTY-OF pointer to STANDING, Since most concepts humans 
discuss are defined in terms of properties rather than anti- 
prop or ties, the MUST version is considered more likely, 
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TABLE 

ACTION OF CONCEPT GENERATOR; EXAMPLE CASE 


onote type 

pointer Involved response 

a-kindhof-chain 

point to intersection 
with model's pointer 

a-kind-of-merge 

- K point to intersection 

with model's pointer 

2„ drop modol's pointer 

negative-satellite 

pair 

drop mo del's pointer 

must-be-satellite 
pair 

retain model's pointer 

must-not-be 

Satellite pair 

contradiction 

supplementary-pointer 
or exit 

negative-satellite or drop model's pointer 

fundamental pointer 
in the model 

negative-satellite or ignore 

fundamental pointer 
in the example 

mus t-h e -5 a te11ite co n tradic tion 

must-not-be-satellite retain mode Vs pointer 


TABLE {cont.) 


ACTION OF CONCEPT GENERATOR; NEAR MISS CASE 


c-note type pointer involved 

response 

a-kind-of-ehain 

1+ If model's node is at 
the end of the chain 
add mus t -no t -be s ate Hite 
2 r if near miss' node is 
at the end of the chain, 
use must-be satellite 
to model *% node 

a-kind-of-merge 

1, replace model's pointer 
by its must-be satellite 

2, rep1 ate model 1 s pointer 
by rtust-Tiot-he satellite 
of near miss 1 pointer 

negative-satellite 

pair 

replace model's pointer by 
its myst-be Satellite 

must-not-be-OAtension 

pair 

reta1n mode1's pointer 

supplementary'pointer fundamental pointer 

in the model 

replace pointer with its 
mu&t-be satellite 

fundamental pointer 
in the near nifss 

insert pointer into the 
using must-not-be satellite 

negative-satelI t te 
in the model 

replace pointer with Its 
must-not-be satellite 

negative-satel 1 ite 
f n the near mi&s 

insert pointer into model 
using must-be satellite 
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5,6 Multiple C -n ot es 

Compar 1 sons, yielding sin-rite c-notes are rare. More 
often, the model tm i Id m &t na ke sense out of a vrhclc oroup 
of c - n c t e s * If the comparison involves a near miss* any one 
of the c-no tos might tie the key to proper node! refinement* 
Moreover, many of the c-notes have alternative 
int erpretat i o-n s that make further demands on executive 
expertise* 

The model builder must therefore consider the c -notes 
and all the possible int erpr eta tiers of each. Then, it must 
produce the set of hypotheses that form the model tree’s 
branches* These in turn must be ranked so that the best ray 
be pursued fir st * 

The case of refinement through an example is simpler 
than through near misses* Since none of the observed 
differences are sufficient to remove the example from the 
class, ft is assumed that all of the differences found act In 
concert to loosen the definition embodied in the model. 
Consequently each c-note can be transformed independent! y and 
a new model generated Ly their combined action. There is no 
problem of deciding if one difference is more important than 
another * 

Consequently, if all the e-notes had but One 
Interpretation, only one now branch would be generated. The 


a - kind “Of-merge c-notc has three possible Interpre tat ions , 
however t and if one such c-note occurs, it is only reasonable 
to create throe branches Instead of just one. The action on 
the other c-notes is the same for all three branches. 

'lear masses cause more severe problems, If two 
differences are found* either of them nay be sufficient to 
cause the sample to he a near miss, while the other 
difference may be equally sufficient or merely 1rrelevant. 

If the -differences have multiple Interpretations nr more than 
two differences occur, the number of possibilities explodes 
and the machine cannot work simply ty generating an 
alternative for each possibility. 

The model builder clearly must decide which 
Interpretation of which differences are most likely to cause 
the near miss. 

The machine first forms two lists: a primary list and a 
secondary list. Each c-note eventually ends up in one list 
or the other. 

Now some c-notes can never make the primary list because 
they are of tiiemselves insufficient to explain why a given 
sample is a near miss. All of these go immediately to the 
secondary c-note list. One example is the situation in which 
a pointer in the near miss corresponds in the current model 
to an instance of the MUST-SATELLITE version of the pointer. 


h m st-satell it e~pa 1r c-note results but certainly is no 
grounds for excluding the near miss from the class since the 
required pointer is in 'fact present* Seme other explanation 
must be fa u nd . 

The next and most obvious my to sort differences is by 
level. This assumes only that the differences nearer the 
origin of the comparison description are the more important* 
This certainly is a reasonable heuristic since a missing 
group of blocks generally impresses a human as be Inn mere 
important than a shape change* which in turn dwarfs a minor 
blemish. Consequently* the program determin es the depth of 
the remain inn c -notes which are nearest the origin of the 
comparison description* All those candidates found at 
greater depth arc relegated to the secondary list. 

The primary e-note list allots quick formation of little 
theories about why the near miss misses and what to do as a 
consequence* These theories are called hypotheses* A 
complete hypothesis specifies one c-note as the sole cause of 
the miss and It further specifies which interpretation of 
that c-note is assumed. Consequently there is a hypothesis 
for each interpretation of each c-note On the primary list* 

The c-note specified as crucial by a hypothesis is 
transformed as if It were the only e-note. The other c- 
notes, both on the secondary and primary lists* are assumed 


by the hypothesis to be tnsu ff ic ient cause -for the near miss. 
Co nse gu entl y as a new model is formulated accord 1n<? to the 
hypothesis* all of the c-notes but one are treated exactly as 
if the near miss were not a miss at all] 

So far a single c-note is assumed to be the exclusive 
cause Of tho miss, Mere all possible com bl hat ions considered 
as well* not only would the branching] increase enormou st y f 
but the ranking of those branches would be difficult. Rather 
than face this, 1 have decided that only one special 
combination of two c -notes is ever permitted to form a 
hypothe sis. 


In this I have exercised what one rsioht Call the 
first heuristic of science; Benin with the linear 
model* the one that assumes all thin os act 
independently; then consider interactions as necessary, 

I next discuss a particular case In which It does seem 
necessary to consider the joint action of two 
differences, It would be unreasonable, however, to try 
for a general method for handling multiple differences. 
In science as a whole, each particular method for 
treating interacting effects Is u sally a major problem 
in Itself and o v er -am bi t iou s search for completely 
general methods is of low utility when premature. 

Further justification for my appproch lies In 
certain observations of Piaget's that Indicate that 
children seem to pay sharp attention to only a single 
feature at any one time [3]- In comparing volumes* for 
example, they use mainly height. Vet* In spite of using 
what appear to be linear comparisons, those same 
children can learn physical concepts with a talent far 
in excess of my goal for this thesis* 

Hypotheses based on two contributing c-notes are added 

to the hypothesis list only when two c-notes with nearly 


151 


identical descriptions occur, Consider figure 5-13, Since 
exactly the same thing characterizes both blocks in the nEar 
miss* there is no particular reason to suppose that one 
difference should be singled cut. C orrse ouen tly a third 
hypothesis is formed, namely that both differences act 
cooperatively. This additional hypothesis takes precedence 
ever the two hypotheses that consider the differences 
separately. It seeps heur i st ica 11 y sound that coincidences 
are significant, The machine creates new models with such 
hypotheses by transforming both of the specified C -notes In 
tho miss-explanation node* 

5^7 Contradictions and Batkina Up 

Gy now one may wander why the pronram should deal with 
alternatives to the r.rain lino of model development at all. 

To be sure, maximum likelihood assumptions nay be vr-ong, but 
then how Could the machine ever know when such a decision is 
an error? The answer is that the main line assumptions nay 
lead to contradiction crises which In turn cause the model 
building pronram to retreat up the tree and attempt model 
development along other branches. 

Consider again the very simple situation presented in 
figure 5-14. The current model and the near miss 
combination generate an a-k ind-of-merge c-nutc for which the 
priority Interpretation is that examples of the concept must 
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be standing, The al ternat iv e, that examples must not be 
lying, causes a side branch in the model devel nnuion t tree, 
but suppose one really wants the concept to exclude lying but 
not insist on standing. Showing the machine the example In 
figure 5-15 doss the job. The tilted trick certainly fs not 
standing and its description has no llftS-bh PPERTY-OF pointer 
to STANDING, let t fie current model has a ’1UST -HfV E-PRD PER TY- 
OF pointer to STANDING. This Is a contradictory situation* 

Vfhen contradictory situations occur., the progran assumes 
it ha s made art incorrect choice somewhere, closes the branch 
to further exploration, and backs up one level to select 
another alternative if any are available there. If no 
alternatives are available, the program backs up still more 
levels Until cither an unexplored alternative is found* or 
the top level is reached. If the top level is reached with 
no other options found, the program succumbs and admits 
failure* flore often an acceptable unexplored alternative Is 
soon found and an effort is made to extend the model tree 
down that branch. Of course, the first alternative a 
contradiction causes to be explored may itself lead to 
contradiction* Back up then starts from the new 
contradiction and proceeds as before* 

In the case at hand, an alternative is found and the 
must-not-be-lying Interpretation of the comparison between 
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the scenes In figure 5*14 leads to a new interned, late model. 
This in turn is refined by the scene of figure E-15 which 
originally caused the contradiction on the former main line, 
No ton trad ict ton occurs on the new path because the ‘-HJ5T*NQT j * 
HAVE-PROPERTY-OF - LYING combination of the intermediate 
model has nothing to clash with tn the example. Indeed the 
new example lends no new information to model development 
along this path, the ^odel being tile sene before and after 
comparison, The new example served solely to terminate 
development of an Improper path In the model development 
tree, 

S.S Other Pa c kina Dp Possibilities 

Many possible refinements to the elementary hackina up 
procedure invite attention. For one thing there are other 
reasons why the learning program might want to back up. In 
addition to the situation of direct contradiction* attention 
should move back op the model tree if there are so many 
differences between the current model and the sample that 
hopeless confusion is suggested. The cause of such confusion 
ts likely to He in the selection of a wrong branch at some 
higher point in the model tree. Similarly retreat is in 
order if t ho program is forced to prooose an unlikely 
explanation to account for observed differences. 

Right now the learning program backs up level hy level, 


blindly exploring all possible paths from One branch point 
before hacking op to the next higher branch point* It would 
be better if attention could neve directly to the noint in 
the tree where the problem began. There a better alternative 
could be elected ami learning could more likely precede In an 
orderly ^ay, 

Certainly the selection of the appropriate point is easy 
In the case of direct contradiction. As explained before, 
these situations occur when some relation is found to be 
essential at some point In model development only to be 
absent from seme subsenuent example of the concept* The 
crucial point is the place where the decision was made that 
the relation was essential. This is the coint whgre 
attention should go and an alternative explanation should he 
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6 Some Server at ed Concepts 
6.1 Physical Models and Functional Models 

In this chapter I explore some of the properties of the 
node! generator through a series of examples. In the course 
of thi$ discussion, words like house, arch* and tent occur 
frequently as they are convenient names for the ideas the 
nachir-e assimilates, He cautioned* however* to avoid 
thinking of these entities in terms of functional 
definitions. To a huEuan, an arch nay be s one t hi no to walk 
through* as well as an appropriate alignment of bricks. And 
certainly, a flat rock servos as a table to a hungry person, 
although far removed fron the Inane the word table usually 
calls to mind. 

But the machine does not yet know anythin? of walking* 
residing, or eating* so the programs discussed here handle 
only some of the physical aspects of these human notions. 

There Is nothing mystical about this. There Is no 
inherent obstacle forbidding the machine to enjoy 
functional understanding. it is a matter of 
generalizing the machine's descriptive ability to acts 
and properties required by those acts. Then chains of 
pointers can link TABLE to FfiOD 1 as well as to the 
physical image of a table, and then the nachine will be 
perfectly happy to draw up its chair to a flat rock with 
the human * given that there is something on that table 
which it wishes to eat, 
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1,2 The House 

Figure 6-1 illustrates what heme means here* Basically 
the scene is just one wedge on top of one brick. But lacking 
human experience, this one picture is insufficient to convey 
much of the notion house to the machine. The model builder 
must he used, and it must be permitted to observe other 
samp Its. 

Suppose the model builder starts with the scene in 
figure 6-1. Then its description generation apparatus 
contributes the network which servos as the first unrefined, 
unembel 1 Ished model, {figure 6 - 6 ) How suppose the scene In 
figure 6-2, a near -miss, Is the next sample. Its not is that 
shown in figure 6-6, The only difference Is the 
supplementary pointer SUPPOPTED-tY* Glancing at the table of 
section 5,5, It is clear that the overall result is 
conversion of the SU PROFITED -6 Y pointer In the old mcdel to 
MUST -BE -SU PPORTEB-BY in the new model. Thus the new mod cl is 
that of figure 6-7. figure 6-fi shows the currant model 
development tree. 

:, luch Is yet to he learned. For one thing, the top 
object certainly must bo a wedge. Shewing the machine the 
near miss of figure 6-3 conveys this point immediately. 
Similarly the near miss of figure 6-4 makes the brick 
property of the bottom object mandatory. But notice that 
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both of these steps cause bifurcation of t he model tree. The 
reason is tha t the machine cannot be completely sure the miss 
occurs because the old property is lost Or because the new 
property is added* The pronran prefers the old-property-f 5- 
lost theory and moves down t lie correspond inn branch unless 
contradicted. In both of these situations* the preferred 
theory is correct resulting In the final model and tree shoi-m 
in fioure 6-9 and finyre G-lb* 

6.3 The Pod estal 

iJev elopmert of a nedestal model proceeds much as -does 
the house with only two essential differences. First* the 
top object must he a brick rather than a ’-Medne* Second* the 
upper object r»u s t: not narry the lower. The scene in 
fiou re 6-11 yields the start in p model. F inure 6-12 forces 
the top object to he £ brick while finure P-13 farces the 
bottom object to he a hrick as well. Figure 6-14 emphasizes 
support, find finally* f inure 6-15 forbids the A 1ARRYS 
rel at ion . 

£. A T he T cn t 

Think of the tent as two redoes* marry inn each other. 

As such it illustrates the hardline of two similar 
differences simultaneously* 

Suppose the be se model is the description of thp scene 
in f inure 6-16 and the first sample is the near miss in 
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figure 6-17* fvio a - kind-of-neroc c-notes result* one from 
each of the two abject 5 because tlioy are tricks net wedges* 
Since they differ only tn source* the hynothes 1s that both 
ACt together is the pr ior it y one, which leads to the result 
In figure 6-19* Now this is complemented by the near miss in 
figure- 6-18 which inf cm 5 the machine of the importance of 
the MARRY 5 relation. Again dual c-no to 5 announce the loss of 
a pair of fiARRYS pointers and twin MUST-MARRY posters are 
installed* (figure 6-26} 

£„ 5 The Arc h 

The arch involves a mixture of the elements seen in the 
previous examples* Eecause of the wider variety of 
differences encountered* it produces a bushy tree and a 
challenge to routines that select priority hypotheses* 

The scene in figure 6 - 21 forms the first model,, 

Con Dining tin's with the scene In figure 6-22* the machine 
deduces that the HAPRYS relations between the top and the 
supports are not crucial* 

Next the near miss of figure 6*23 indicates that the 
Support relations are crucial, Derain* both new MUST - BE— 

3U PPORTED-BY pointers are handled jointly, and are installed 
at once. 

The machine learns perhaps the most important fact from 
the near miss in figure 6-24* 


Here the two supports touch* 
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su pplying two ' 1A RR Y 5 pointers to the d escript icn» This 
cannot be allowed. Resooncinq, the machine in sort S MU ST-HOT- 
MARRY pointers between the two supports In the frto<i el * 

Some way think that in asserting the MU ST-fJOT-^ARRY 
relations* the machine overlooks what they consider the 
real principle, that of a hole or passage* But for a 
child building with blocks, to have a hole and to have 
two non-touehl ng supports are very nearly the same idea* 
Consequently the machine 1 s opinion seems adequate for 
the moment. Experiments such as those may help to 
expose exactly what kinds of network relations are 
adequate for a model of human thinking, from infant to 
adult. 

Finally, the top object is not necessarily a brick. The 
sample in figure 6-25 teaches the machine that anything In 
the class OBJECT will do, since OBJECT Hes hut one step 
removed by an A - KIKU - OF pointer f^om both WEP^E and ERICK* 

6*6 The Wedge 

The capabilities of the model builder certainly extend 
beyond the level of object configurations, whose descriptions 
allow the machine to learn about scenes, here the 
development of the wedge model illustrates the point, 

Given the wedge in figure 6-26, the description 
generated is that of figure 6-27. 

t\ ex t, comparison with a br k establishes a MU 51-BE-A- 
KlfiD-OF pointer to TRIANGLE, (figure 6-22) 

But now suppose the partly occluded object in 
figure 6-2 9 Is comoared first with the current model of wedge 
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and I lien with a descrf pt Ion of a brick* As figure 6-30 
illustrates* the surprising result Is that both camperisens 
have nearly the same descriptions. In both cases two 
rectangles arc matched and a third side left unmatched. In 
one case the unmatched side Is another rectanole* and in the 
other, it Is a triangle, But there is aS yet no way to 
prefer one match above the other! 

The problem Is a 1 ft tie Involved. Ho severe mismatch is 
evident to the difference description evaluation program 
because the '1UST-EE-A-KIND-DF pointer is anchored to a node 
that Is not matched* The model as it stands asserts firmly 
that if three sides are seen* one of them must he a triangle* 
but it does not assert that such a third side must be 
pr e se n t * 

This may seem to be a bug at first* but the problem is 
really the machine's lack of experience. So far only two 
configurations have contributed to the node 1 , Consider the 
result of refining the model with the scene in figure 6-29 
which Is causing the trouble. It Is a near miss, Sut the 
difference between it and the current model Is an exit c-note 
to the triangular side. The modEl builder perceives the need 
for an emphatic pointer and 0*1 E- PART -MU ST-BE is inserted, 

(f igure 6-31 ) 

Now if this model is compared again with the scene that 
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just refined the model, the one fn fioure 6-29, the result Is 
an exit c-note bearing an emphatic pointer, fN E-PAR T«*"t ST-RE, 
Swell a e-note strongly suoqests bad match to the evaluation 
program, and the apparent Inadequacy disappears, 

The scene in figure 6-32 establishes a fin^l refinement. 
This wedge shows only two sides. After a bit of thinking, 
the program decides one of the two rectangular sides is 
optional and produces its final model, [figure 6-33) 

6,7 The Composite Column 

When a concept Involves groups of objects, the model 
generation problem really is no nere difficult. It just 
becomes a matter of concent rat In a on relationships between 
the typical numbers of the groups studied. 

Consider the notion of the composite column, hereafter 
referred to simply as the column. Figure £-34 shous such a 
column and figure 6-40 shows part of the corf esp ond inn 
descriptive network, Ttifs tic sen lot ton is gradually 
transformed; into a reasonable model in the following tray: 

Figure 6-35, with its bricks askew but otherwise the 
same, introduces the MU ST-MARRY pointer, Moure 6-36, made 
of wedges Instead of bricks, relaxes the fncllnatfnr, toward 
bricks, And figure 6-3 7 causes replacement pf H!l FPffiTF?]~IJY 
by HU ST-BE-SU PPORTED -BY. 

Heat, figure 6-38 contributes the most important part of 
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the model, the part that demands a group. The scene In 
figure 6-33 consists of only two objects and Is not grouped 
because the grouping program requires a minimum of three 
objects* The resulting c-notes reflect the mandatory need 
for a group by way of a Oh E—P ART -f5li ST *BE pointer to the node 
representing the group. 

Finally* figure 6*39 generalizes the model In an 
Important way because its typical-member node differs from 
the model's principally because a different number of objects 
Is indicated. Itl one case the NLiM &Eft *0F BER5 pointer 
points to 3; In the other, 4, But since both 3 and 4 are 
Integers arid have A-KIND-OF pointers to INTEGER.* when the 
comparison between the two is made an 3 - Rind-of-merge c*note 
results. The next model consequently has a NIW BER-OF -MEMCERS 
pointer to INTEGER, rather than 3. Mow columns may have any 
number of objects greater than two. 

Figure 6*41 displays the overall result, 

*3 The Arcade 

The problem of learning about the arcade shown in 
figure 6-43 adds another interesting dimension to the n*cdel 
generation problem* Nothing new is needed in developing a 
sequence of models for the arcade, with one Important 
exc ept io n: 

The arcade is a conglomeration of substructures* rather 
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than of simp 1 y objects. As such the arcade development Shows 
how the model builder can appeal to things already learned in 
the process of understanding nor? complex structures. 

In build ing a description of an arcade, the description 
programs identify arches using the previously assimilated 
arch model. This leads to the description partially shown In 
figure 6-43. Then subsequent samples, shown in figure 6-44, 
figure 6-45, and figure 6-46 inform the machine that there 
must be a group, that the group elements rust be arches, and 


that the relation must he IN-FRQH -OF* 


The deduction that one arch is in front of another 
involves methods less encored and less sound than 
techniques previously described for dealing with 
individual obiccts. Indeed this Is a virgin field of 
inquiry that l have thought about only enough to write 
nroc-rams which can handle these few examples* It ^ L 
K Hr f 0 r example, if each structure will require its 
own set of heuristics for determining inter-greu p 
relations, or if general principles can be enunc ated* 

So far my pripitive programs assume only the following 

rules: 

1 if there is a chain of support relations 
between every object in structure A to some ob.iect 
in structure B, the C can be said to support ij ., 


2. If sccie object In A Is 
in B, but not vice versa, 
in frofit of K. 


in front of some object 
then A. can be sa id to ho 


£.5 T he Ta ble- 

The table is much like previous examples except that 
grouping is done on the basis of abject form and function, 
rather than relation chains. 
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gtudy the table In figure 6-47 and the description, in 
figure 6-48, The essential features of the table are 
Introduced by the following sequence of steps: 

First the table should have bricks for legs. This idea 
is easily conveyed by the non-table of figure 6-49, 

[iarecver* this concent fori of table excludes structures such 
35 that In figure fi-50, a fact which is handily incorporated 
through a HU ST -NOT-HARR IT pointer. Next, since the non-table 
in figure 6-51 has only two supports, no grouping occurs, 
which leads to insistence on a group In the next model 
refinement. This entirely parallels the process by which 
the column was found to involve a group. Finally, the sc$ne 
In ff cure 6-5 2 leads to replacement of the SUPPORTED-BY 
pointer by MU ST - RE -SU Pp-ORTED-SY. Figure 6-53 shows the last 
nod el in this development, 

6,10 7 he Arc h in Uept h 

So far the illustrations have shown networks only to 
that depth appropriate for understanding. Figure 6-54 shows 
the model for the arch In somewhat fuller bloom and better 
Indicates the breadth pf the information available to 
programs that use the model. 
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b.11 Direct ion. 5 for* improvement 

Experiments with the concept generator titillate rather 
than satisfy, for each success suggests new ideas to be 
explored. As It stands, the concept generator Is a healthy 
baby but not a contr ibu t ing adult* The possibilities are 
staggering, however, 

fine way to improve no del generation abilities is self^ 
evident! the system improves as the ability to describe 
improves, The heuristic rtetern inat Inn of support and 
occlusion and the rest can be inprnved, and more important, 
other so far neglected relations and properties lie 
unexplored. Description programs should and can be taught 
to differentiate between cubes, bricks, boards, and sheets. 
They should know in general when the terms lying and standing 
are meaningful * They should bo able to relate structures 
more carefully. They should present alternative descriptions 
If situations are plainly ambiguous. They should know about 
color ar.d texture* They should know about holes* 

Dpt as description becomes Letter, the burden on the 
model builder becomes greater, for the proliferation of 
properties and relations means that the nunber of c-notns 
multiplies, thereby complicating all of the decision 
processes. To cope, ft will probably be necessary to 
Institute further techniques for location the centrally 
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impgrta rut c -notes, 

For one thing, c-notes involving certain pointers might 
rate priority attention. These might* for example, be 
pointers that frequently played central roles In the 
development of other models in the past* Additionally, some 
point Or s might be tentatively convertod to MU ST-BE versions 
by virtue of frequent occurrence in the samples under 
consideration* Previous objections to this idea still stand, 
tut done 1*1 th care, with the assumptions behind the action 
somehow indicated* sene good might cone from this* 

If the machine could ask the teacher nuest ions, it would 
open up another powerful kind of finesse* In situation* 
becoming precariously ambiguous, the model builder would a Sic. 
directly if some relation is important, or per bans display 
several imagined scenes to the teacher, requesting a 
statement about which are in the class to be learned* 

Finally, and perhaps crucial, some confrontation needs 
to be made with function* If ha t to dp* however, is unclear* 

In dealing with the table, the development was already 
strained, It is not really adequate to think of legs solely 
as standing bricks, and genera 1 izat Ion to the class of all 
objects seems specious. So far all models and 
identifications deal only with descriptions of concepts' 
parts, but this is not adequate to handle the notion of the 


leg* Hhat ts needed is the id ea that s one t hi n et is b leg, not 
because of what its parts are and hew they arc related, but 
rather because that something relates to srniethfng else in a 
particular way, namely through the SUPPORTED-BY re latter* 
Given the ability to think this way, programs could identify 
round legs, square legs, or legs made of several parts. 
Functional use would serve as a very powerful Interpretive 
and grouping tool adding Immeasurably to the limited 
understanding now attained, 

I emphasize that no fundamental barrier prevents 
programs from thinking functionally* Indeed some 
possibly useful programs already exist. As work 
progresses, further analysis dan be attempted and 
identification can be expanded to include the 
relationships that suggest function. At first work 
should concentrate on things that are defined In terns 
of currently observed use, rather than things that are 
defined In terms of conjectured potential use. A table 
leg, for example, is a leg because it currently is 
observed to support something. The same object would bo 
far loss likely to be identified by a human as a leg 
were it seen separated from any table* There is little 
possibility of identifying something as a table leg on 
the basis of potential use unless a leg Is specifically 
searched for. 


7 Identification 

7*T Matching and Identification Alternatives 

Once there are programs that describe scenes, compare 

description networks,, and by lid models, one may go on to 

using these programs as elements in a variety of other goal- 

oriented programs* The probl em*solv ing programs described in 

this chapter have the following kind of responsibilities: 

To see If two scenes are identical. 

To contrast two scenes and report the differences 
between them, roughly in order of Importance* This 
supplies Information that may prove useful to programs 
that use a mechanical hand and arm to build copies of 
SC en e s, 

To compare some scene with a list of models and report 
the most acceptable match. This is the identification 
problem in its simplest form. 

To identify some particular subset of the objects in a 
scene, This is not the same as identifying an entire 
scene hecausE important properties may be hidden and 
because context may make some identifications more 
probable than others* 

To find instances of some particular model In a scene, 

It Is frequently the case that the presence of some 
configuration can, he confirmed even though ft would not 
be found in the ordinary course of Scene description. 
This requires the ability to discern groups with the 
required properties In spite of a shroud of Irrelevant 
and distracting information. It Is not uni ike the 
probl Em of finding the bunny on a Playboy cover* 


lDO 


7,2 Exact. Match 

If two scents are Identical* then the networks 
describing those scenes must be Isomorphic. The nodes of the 
two networks must relate with each other in the same ways, 
anti the nodes must relate to general concepts such as fid ICK 
and STANDING in the same ways. Consequently, comparing two 
such networks produces a simp lr kind of comparison 
description. There is a skeleton* which indicates how the 
parts of the scenes interrelate* and there is a group of 
intersection C-nOtOS that describe how the parts of the scone 
are anchored to the general store of concepts. None of the 
other types of c-notes appear because identical scenes cannot 
produce two networks with the necessary aberrations of form* 

Conversely, if comparison of two networks results in 
intersection c-notes only, then the parent scenes must be 
identical in the sense that the description generating 
mec han isms employed produce exactly matching networks. There 
can bo variation* hut nothing so great as to vary the action 
of the description generator. The scenes in finure 7-1 are 
identical with respect to the descriptive power of my 
programs because in both cases the relations observed are 
LEFT-OF and RIGHT-OF. More capable programs night conn la in 
that FAR -TO-T d£-LEFT~QF and FAR-TO-THE -ft IGliT -OF hold in one 
scone* while only LEFT-OF and RIGHT-OF held in tho Other. 
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The scenes are clearly not Identical with respect to a 
program with such a capability* 

One gen eral izat ion adds a degree of flexibility to this 
procedure that humans seem to exercise* The kind of question 
to be answered is not of the simple form "Are two given 
scenes identical?" but rather, '"Are two given scenes 
identical with respect to a certain degree of detail?" 

An approach to this new problem Is to modify the 
Inter section-only criterion* Instead of reguiring all c- 
notes to be intersection c-notes, one requires that all c- 
notes be Intersection c-notes at or above a certain level. 
Thus if level one is specified* then the scenes in figure 7-2 
are indeed identical because the comparison description shows 
nothing but intersection c-notes at or above that level, 
{figure 7-3) Cut i f level two is also of concern, then the 
scenes are not Identical because an a-kinrf-of-me^ge c*note 
describing the difference in shape between face t and face C 1 
appears on that level* 

This use of level plainly defines a concrete substitute 
for the otherwise vague notion of dooree of detail* One 
simply says two scenes either are or are not identical down 
to some particular level and the existence of non- 
inter se ct i an type c-notes beyond that level is net of 


concern. 
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7.3 H 1 gr c s s i on 

The exact watch detector is a major part of a curiously 
simple program that checks for a certain kind of left-rinht 
synroe try* The method ts as follows: 

T. Copy the- description of the scene exactly* 

Z. Convert all LEFT-OF pointers in the copy to E IGFIT-OF , 
and all Fl IGHT *0F pointers to LEFT-OF. 

3, Compare the original description against the modified 
copy. If the natch is exact, the scene is symmetric. 

This is* of course* an abstraction of the familiar 
condition for y-ax i S symmetry in the ma tiiemat tea 1 sense* 
whereby symmetry Is confirmed if a ltd only if for every point 
In the scene, (x T y), the point {-x,y} is also in the scene. 
Switching LETT-OF and RIOKT-CF pointers is the analogue of x- 
coordinate negation and network matching corresnonds to a 
check for invariance. 

To sec how this works, consider the scene In figure 7-4. 
The center object, A a is flanked by B on the left and by C on 
the right. Figure 7-5 shows the resulting de scr ipt 1 Ob. 

There are nodes corresponding to objects A, 5, and C , and 
there are LEFT-OF and ftlftHT-CF pointers Indicating their 
relat Ion ships. 

Figure 7-6 shows the copy of the network with the LEFT- 
OF and RIGHT-QF pointers switched. Notice that the original 
network and the copy are identical. Flode A matches with S'* 
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C with C ' * and B with A 1 , Since there are no differences, 
the rnachine cep eludes the scene is in fact synmetr it. 

Of course, a nonor a 1 liat ion of symmetry is possible, 
just as generalization of identity is. The nachfne need only 
perform the same copying and exchange operations and then 
check f or non-inter sec ticn c-notes only to sons particular 
depth. Thus the scene in figure 7-7 is symmetric to depth 
one because the groups of objects arc synmetr ica 11 y placed. 

It is not symmetric to depth two* however* because the 
placement of objects within the groups is wrong* The scene 
in figure 7-8 differs from that In figure 7-7 because there 
is not only symmetry in the location of groups of objects, 
there is also symmetry in the placement of objects within the 
groups* This means that the scons is more $ynnetr ic to the 
machine In the sense that the symmetry detection program 
remains happy at a deeper level of inquiry. 

The machine knows LEFT-OF and RIGHT-OF are opposites 
because information about these pointers lies in the general 
memory net. (figure 7«-9} Consequently, It Is unnecessary to 
tell the program explicitly to substitute RT'iHl-Of for tEFT- 
p art! vice-versa. One- need only ask the s ynrn e t ry pr o n r a n If 
there is symmetry with respect to cither the pointer L, EFT-QF 
or JlIGIfT^OF. The machine itself can conjure up the 
appropriate substitutions by working throunh the OPPOSITE: 
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pointer from whichever re lat ton Is su nr I led T be it LEFT-OF or 
RIGHT-OF, Similarly, if ore asks for syndic try with respect 
to ABOVE* the program realizes that the proper substitutions 
are BELOW for ABOVE antl ABOVE for BELOW, IH-FRONT-OF gives 
BEHIND for IN-FRO NT -OF and vice-versa for a somewhat unusual 
Lind Of symmetry question. 

Certa1nly mIxtures are also possible. One can ask for 
Noth left-right and above-belay substitutions which Is an 
abstraction of symmetry with respect to a point, 

Mathematically speaking* there is sy mire try with 
respect to the particular point (0*0) if for every point 
(x*y) in the scene, there is also a point (-x*-y). The 
way to check a drawing is to imanin-e moving every point 
straight through the origin until It Is again the same 
distance fron- the origin but lies in Ute opposite 
quadrant. For example* point P in the symmetric drawing 
in figure 7*10 goes to point P'. when the end result is 
the same as the original drawing, then the drawing is 
symmetr ic. 

More important ts & combination of a left-right and an 
in-front-nf -- behind switch. This one gives the machine a 
chance of realizing that two scenes are just front and hack 
views of the sane configuration as are the scenes in 
figure 7-11, 

Eventually I think the machine can come Upon the 
symmetry notion in the same way It now learns about 
arches and houses, Rut at this point I do not think 
there is enough comparison describing capability. The 
needed step is the introduction of a program that 
■generates global c -notes from the local ones already at 
hand, thereby introducing the kind of hierarchy into the 
comparison d escript ion s that is already the standard in 
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scene descriptions. One obvious ability of such a 
pr ogram would be that -of noticing a preponderance of 
similar c-notes, I think that this and sons of the 
double comparison ideas proven useful in doing anaToay 
problems are just the things the machine needs to learn 
about symmetry, 

7.4 Elementary Identification 

Suppose a scone Is to be identified. If possible, as a 
II1XJSE, PEbESTAL* TENT, or ASCII* The obvious procedure Is to 
match its description against these for each of the node Is 
and then somehow determine which of the four resulting 
difference descriptions Implies the best match. 

Recall that models generally contain ngst-be satellites 
and must-not-be satellites while ordinary descriptions do 
not. Consequently, comparing an ordinary description against 
a model leads to a variety of c-notes not found when ordinary 
descriptions are compared, Among those are mu st-bo-sate 11 ft e 
pairs, mu st -not - be-sa tel 1 ite pairs, and various flavors of 
exits and supplementary-pointers. Such c-notes are decisive 
in the Identification process, 

Consider the case where some pointer in a scene's 
description corresponds to its nust-not-be satellite in the 
model, This clearly means a relation is present that the 
model specifically forbids. The resulting must-not-be- 
satellite-pair c-note In the difference network is such a 
serious association impediment that identification of the 


unknown; with the model is rejected outright without further 
€ on 5 id er at ton + This means that the near-arch in figure 7—12 
cannot be identified as an arch because the network 
describing the near-arch has ’lflRRYS pointers between the two 
supports while the model has 'fU 5T-NOT-MARRY pointers in the 
same place* The combination produces a comparison 
description with a mjst-not-be-satell ite-pa 1r c-nnte that 
positively prevents natch* 

Identification with a particular model is also rejected 
if the difference description contains exits or 
su pp lementary-pninter c-notes which Involve must-be 
satellites* Such c-notes occur when essential relations or 
properties are missing in the unknown. Thus the two bricks 
in figure 7-13 do not form a pedestal because the model for 
the pedestal has a MUST-GE-SUPPORTED-EY pointer where the 
scene of figure 7-13 has nothing. The result is a 
suppl snentary-po inter c-note involving the must-be satellite 
MU ST-EE-SU PRORTED-EY. Again match fails. 

The case of a - kind-of-meree c-notos involves a slightly 
more complicated rule* Recall that merge c-notes occur 
generally when two nodes stare properties that are not 
identical „ but which fall into the same general class* The 
situation must be one where two linked nodes exhibit closely 
related pointers to two other nodes from which paths of A* 
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KIN3-QF poi inters lead to a common inter sect ion * Figure 7-14 
shows such a s itua tlon* In this case the- unknown Is a kind 
of wedge while the corresponding object in the nod el must be 
a kind of brick* Goth HEDGE and GH1CK are kinds of objects,, 
which directly leads, to a merge c-note associated with a 
Mil ST-BE-A-KIND-OF pointer in the model. But the fact that 
the unknown has a property in the same class as a property 
required by the model Is insufficient. To insure rejection 
of such matches* the rule is; Refuse identification if the 
model's pointer contributing to the merge c-nnto is a must- 
be-sat el 1 fte* 

Figure 7-1 E summar izes the procedure used on each c- 

note* 

Match of the scene in figure 7-1 & against the PEDESTAL, 
the TEHT, and, the ARCH all lead to difference! descriptions 
with c-notes that forbid identification* The PEDESTAL falls 
because a marge indkaf.es that the required A-FIND-OF 
relation between the top object and BRICK is missing* The 
TENT similarly fails because both of its objects must be 
wedges. The ARCH fails because the model has a MUST-EE- 
SUPPOTiTED-EiY pointer to an object missing in the unknown. 
This In turn causes a fatal e* it c-note in the difference 
d escription, 

Of course the rr a chine can also match the sample 
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pedestal, tent, and arch of figure 7-17, figure 7-18, arid 
figure 7-19 against the same list of models. It makes the 
correct ident if ica ti on in each case. 
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1 he next problem emerges because some unknown nay 
acceptably match more than one model In a trial list, 

Suppose one defines a new sort of arch that is just like the 
old arch except that the top object must be a wedge rather 
than just any object* Call this new model the WEDGE-ARCH. 
Then the scene in figure 7-2G certa ini y matche s with both the 
OftD INARY-ARCIE and the new 1{EDGE-ARCW, There is only one 
slight variation In the difference descriptions. In the 
HEDGE-ARCH case, one has a must-be-satellfte-pa.fr c-note 
because the unknown has an, A ^ KIN p - 0 T pointer to WEDGE and the 
model has a F'UST-BE-A-KIHD-OF pointer. In the ORD IN AR V-ARC H 
ca.se t there fs sfmply an A-KIND-OF pointer from the model to 
ABJECT, which with the unknown's A-K1I7IT-0F pointer to WEDGE 
forms an a-k ind -of-mer ge c-note. 

Of course there is nothing really wrong with reporting 
both ORD INART-ARCH and lOGE-ARCli as the identification of 
the unknown. Still, given several possible identifications, 
there should be some way Of ordering them sgcli that one could 
be reported to be best in some sense. To do this I associate 
each kind of difference with a number and combine the numbers 
to form a score for each comparison. Figure 7-21 shows the 
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scale associating difference types hv i t *1 numbers* It evolves 
hour i st ica 11 y from observations 1 i ft.e the ^ollowino: 

1. The intersection c^ncte has an assigned value of 1 . 
This anchors the scale as all other numbers are fixed 
according to how (rood or bad the c or re sound inn c-note 
seems relative to the intersection c-note, 

2 . A mu st-be-sa tel 1 Ite-pa 1r c^note suggests good match 
even more strongly thou the intersection because It 
indicates that relations are present that are known 
to Le essential, A value of 3 dives it three tines 
the -weight of a slmMe Intersection, 

3* Exit or su ppl enientary-poin tor c-notes that involve 
mu st - be - sat ell ites are distinctively Lad because they 
indicate vital properties are miss inn. The value is 
a tiamaginc -5, 

4, Other exit and supplementary-pointer c-notes are bad 
but not nearly so bad, A score of -2 seems about 
right , 


5, In st -n ot - be - sa tel Mte pairs are very had evidence 
Indeed. The worst score of -5 is deserved, 

6* The a - k ind-of-merge is nesitivc or negative dependlno 
cn whether either of the pointers are must-be 
satellites* If a must-be satellite is involved, an 
important property is miss-inn, resulting in a -4, 
Otherwise, it indicates lor.so association, not as 
tight as that announced by an intersection, A ,5 is 
u sed. 

Once differences are noted and number associations 
are made, a program must combine the numbers in a 
reasonable way* If SC OR El U : M] represents the score of 
comparing yn.known U against model 1 ’ t then 1 use 
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SCORED) :M] = W(1 )H (1 ) + +U(n jN(n) 

Hirer e 

M(i ) =■ exp[-L{ i)] 

5 nd 

N[1) is the number associated with the 1th 
tfiffcrence, 

W (i) Is the weighting factor that reduces the 
influence of Tower level differences. 

L£ i) is the level of the 1th difference* 

Combining thE terms additive'ly is convenient, and the 
weighting ternts* the , handily reduce the Influence of 
the lower level d iff erenc es„ I have no stronger reasons 
for usinn this linear formula, and It is not Sometliinn to 
he defended to the death. Gut I do net think it would nay 
to put much effort into tuning such a formula because more 
knowledge about the priorities o* differences should lead 
to far better programs that do rot use numbers at all* 

7,5 Identification in Context 

Examine figure 7-22. ’lot fee that object 6 seems to 
be a brick while object 0 seems to bo a wedge. This fs 
curious because D and D show exactly the same arrangement 
of line5 and faces. The result also seems at odds with 
the machine's models and identification process, as 
do serf Led sc far, because so far snytMnq identified as a 
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wedge must fra vs a triangular faco. 

(Jut rtf course context Is the ex planat ion , different 
rules nust be used when programs try to identify objects 
or groups of objects that are only parts of scenes, rather 
than the whole scene. In the case where the question Is 
whether or net the whole scone ca n be Identified as a 
particular model, It is reasonable to fnsfst that all 
relations deemed essential by the model be present, while 
a VI those forbidden, be absent. But when the question is 
whether or not a few parts of a scene can be identified as 
a particular model, then there is the possibility that 
some Important part may be obscured by other objects. In 
these situations, my identif lea t ion propram uses two 
speciat heur1 St 1c s: 

First, the coincidence of objects lying In a lino 
seems to suggest, that each object is the same type as the 
one obscuring it unless there Is toed reason to reject 
this hypothesis* This is what suggests object. D Is a 
Liedoe In figure 7-22. 

Second, essential pro nor tics in the node! may ho 
absent In the unknown because the parts Involved are 
hidden. This Is why Identification of object D with wedge 
works even though 0 lacks the otherwise essential 
triangular face. The reouirenent that forbidden 
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pro pert ie s do net occur remains in force, however* 

Elaborate work can be dene on the problem of dec id inn 
if the omission of a particular feature of some mode! is 
admissable in any particular situation, ^ly program takes 
a singularly crude view and Ignores all omissions, 
fi ejection of the hypothesis that the obscured is like the 
obscurer happens only if the machine notices details 
specifically forbidden by relations in the model. Thus 
the effort is not to select'the test matching model, but 
only to verify that a particular identification is not 
contradictory. This means that object b in figure 7-21 
is confirmed to be brick-like while brick-ness Is denied 
to D because of the ruinous apparent triangularity of the 
side face. 

Cf course if the propagation of a property like 
brick-ness or wodge-ness down a series of objects is 
interrupted, then the unknown must be compared 1 with a 
battery of models, with the program still forgiving 
onti ss ion s, but now searching for the best of many possible 
fdent if 1 cat 1 on s* 

to matter what the method by which a partly obscured 
object is identified, the use of a PR 0 6 A B L V —A - KIND—OF 
pointer instead of the basic A-KIWD*0F is used to qualify 
the conjectured relationship between the object and the 


nod el It is Identified with* 

Fitrure 7 -?4 shows how the various pieces of this 
procedure fit together! and figure 7^25 shows what happens 
when ft moves down a simple row of objects, 

7.6 Learning from Mistakes 

Suppose the program attempts to identify the scene in 
figure 7-2 6 as a pedestal, Identification fails because 
tho resulting tyne of a-kind-of-merge c-note cannot bo 
to 1 orated, Still it would be a pity to throw away the 
information about why tho natch failed, instead the 
otherwise wasted matching effort can be used to soonest 
new identification candidates. 

The way this works is quite simple. First the 
machine spends idle time corn par inn the various models in 
its a rma men to r iiim with each other, whenever the number of 
differences observed are few, a simnlified description of 
those differences is stored. Thus the machine knows that 
a house is similar to a pedestal, from which it differs 
only in the nature of the top object. 

These descriptions link the known models together in 
a sort of similarity network, {figure 7-27] 

This network and the difference descriptions noted In 
the course of identification failure help decide what 
model should be tried next. The description describing 
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tlie d, If f erfhc es. between an unknown a nd a particular model 
is compared talth the descriptions of the similarity Pet, 

If the difference between the urtt.nown and a particular 
model matches the difference between that model and some 
other model, then identification: with that other model is 
I i kel y, 

For example, the scene of figure 7-2E relates to the 
model of a pedestal In roughly the same way that the model 
of a house relates to the model of a pedestal, house is 
consequently elevated to the top o F the list of trial 
models. Figure 7*£u clarifies the procedure, 

7 , 6, . Similarity Descr lotion 5 

The similarity descriptions are simplifications of 
the comparison descriptions and are part of the 
description of each pointer that ralates similar concepts, 
When a losing identification reminds the prooram of some 
difference structure it has seen, no serious commftment is 
made and mistaken conjectures do not hurt much. 

Consequent! y it is desirable to strip t fie difference 
descriptions to the important el en e in t s , thereby saving 
storage space and increasing matching speed, even though 
some wrong models may be proposed as ]ikoly 
identifications. The simplified description therefore 
consists in part of a sort of skeleton, [figure 7-29) 
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Intersection c-notes and others associated with positive 
numbers cm the evaluation scale are ignored because only 
the disruptive c-notes are of interest here* These 
disruptive c-notcs, which suopest poor match, are hunn on 
the skel eton . 

Figure 7-30 shows the simplified difference 
description resulting from Comparison of the house model 
with the pedestal model. Notice that it is exactly the 
sane under this simplification transformation as that 
resulting from comparison of the oedestal model and the 
scene in figure 7-26. 

7.6.2 Ci ef in. ft ion of ■Quantitatively Small 

This whole similarity scheme depends on the fact that 
two models may have only ore or a few differences that 
make them stronaly different In a qualitative sense. 
Indeed, the similarity links should exist only when the 
models involved a^e reasonably close in the sense of 
producing f ev: d iff eronce s. When this is true, an unknown 
that nearly identifies with one model in the sense of few 
differences is assured pf matching well with the cther ( 
particularly when the two sets of d Iffarentes natch. Thus 
there must be some rule for dec id top if the number of 
differences Is sufficiently small to warrant a pair of 
Fo inters in the similarity network. Currently the machine 
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considers sufficiently smell to mean the number of 
mismatch-causing c-no-tes is either less than tv/o cr less 
than onE-third of the number of other c-notes, 

7.7 The Needle fn the Haystack 

The scene of figure 7-31 is curious in that one can 
find an arch* a pedestal* a house* and a tent in it if one 
is looking for them. Cut if they are not specifically 
searched for, mention of these particular models Is 
unlikely to appear In a description of the scene. 

Although the cqnf f gurat ions are present, they Ore hidden 
by extraneous objects so veil that general grouping 
programs are unlikely to sort them out. Yet the Question, 
"Does a certain model appear in the scene? 11 is certainly a 
reasonable one. One way to attack it divides nicely Into 
three partst 

1* Find those objects In the scene that have the best 
chance of being identified with the model, If the 
model has unusual pointers or references unusual 
concepts* the program pays particular attention to 
them. Similarly* extra attention is paid to the 
emphasised parts of the model, for if mates cannot 
be established far them* solid identification 
cannot be affirmed, Happily* my standard network 
matching program dees the se things without 
augmentation. The result is a set of links 
between the objects of the model and their nearest 
analogues in the scene. The other parts of the 
sc en e remain unlinked and end up appearing in exit 
c-notes* 

2* Once a good group of objects is picked, then the 
pointers relating these objects to the other 
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objects in the scene are temporarily forgotten* 

In human terms* this Is like painting the subgrouo 
a special color or otherwise reducing possible 
confusion from relations with the surround In □ 
objet ts, 

3* Finally, with the best group of objects set into 
relief by the pr e viou s excision* the ordinary 
id en t if icaton routines are applied with the 
expectation of reasonable perf otrance * 

"he folly of direct application of the identification 

programs lies In the myriad Irrelevant exit c-notes that 

the extra objects fn the scene would cause. Such clutter 

leaves the machine as bewildered as it does humans* 

7*S Reacting to Identification 

Qnce identification of a substructure happens, the 
discovery should contribute to the storo pf knowledge. 
Figure 7-32 illustrates some of the more obvious things 
done after identification of the house and arch in 
figure 7-33, The highest level node no longer connects 
directly to the individual objects. Instead* those 
objects dangle from new sub scene nodes by ONE-PART - IS 
pointers. Similarly the old top level concept points to 
the new subscenes with ONE-PART-IS, The Sub scene nodes 
naturally point by A-KIND-OF to the models they identify 
in t h, 

Rather more can tie done if the machine knows 
someth fnq about hovi the relations of a group's components 
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to cm terra 1 objects dictate the relations of the or cup* 

Any knowledgeable machine knows that a boose confiscation 

rests on whatever its Lott on object rests On* More 

generally, the following rule seems reasonable: 

Supocse A and B are groups of objects identified as 
substructures, then if an object of A relates to an 
object of H by SU PROP TED -BY , then so bstruc ture A 
relates to substructure & by £U PP'OfiT Ef] -B Y, 

In consequence, the net in figure 7 -3S becomes that In 
figure 1 -34, 

7 * S , t tx ambles 

Using this same procedure-* the scene tn fiouro 7-35 
soon reaches the state of i 11 uroinat Ion sh-own fn 
figure 7-37* By examining either the picture or the net, 
it is easy to see that the arches AT, AL, and A 1 ! 
themselves constitute a sort of super-arch with arches as 
parts instead of objects. The mac bine does not refuse 
this substitution since the model for ARCH has only ONE- 
PART-IS pointers to BRICK and OBJECT, not OH E-PART-MU5T-DE 
pointers. The matching score is sfnply lower than It 
would be for arches made of bricks. The final description 
essentially states that the scene consists of « sort of 
arch supported by an arch composed of three arches. 

Figure 7-36 shows a richer example including 
instances of a pedestal* Again the machine Identifies 
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■groups* establishes relations between the Identified 
groups* and then tries to identify groups of groups* 
reporting eventually that there is an arch composed of an 
arch on top with two pedestals for supports* It then 
notices that thts generalized arch is supported by two 
ordinary arches* But the generalized arch on ton of two 
supporting arches again is a kind of arch* the fifth and 
last discovered* 

7,3*? Other Relations 

1 have not thought much about the calculation of 
other group properties. It seems roascnable, however, 
that a set of programs using the following ideas should 
work to some extent* albeit crudely. To find Ft!-FRONT-OF 
relations between qroups one car use the above rule for 
SUPPORT with the obvious exchange of IN-FROUT-QF for 
SUPPORTED-BY. To establish the size of a oroup one can 
add together the individual areas of its objects. To 
check for LEFT-OF and RIGHT-OF, one uses the center of 
area and extreme points of the entire group rather than 
those of an individual object* But otherwise the left- 
right algorithm may remain the sane. 


3 Closing Roarks 


8 J A Sy st ero 

The flow diagram in figure 8-1 shows how the 
techniques fit together with those of others to form a 
primitive seen e^perte 1v ing system. At the very beginning 
lies the scene, from which all Information ultimately 
derives, A program developed by Griffith [7] watches the 
scene through an eye resembl ing television camera* The 
result is a line drawing, iicxt programs of M a ha bale [T] 
arid Guzman [2] classify vertexes and group regions Into 
bodies. Next is a stage in which object identification 
is done, Following closely, one has the determination of 
object-object relations end then group identification. 
Finally there Is identification of group-group relations, 

Gcyond this* action depends on intent* On one path 
one finds attempted identification of the entire scene 
with a known model or models, Cn another, an effort is 
made to find an instance of some particular model in the 
scene, still another path involves use of the 
description to help form new concepts, 

3 .2 Cone In sions 

This collection of ideas and techniques supports four 
major contentions* each of which depends on these 
preceding it. These things are small steps for a man * but 
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*1 1 an t leaps for a machine, 

1* A computer can produce a detailed scene 

description consfstlna of the same sort of facts 
humans observe* 

2* These descriptions lead in turn to descriptions of 
how scenes compare with one- another* 

3, An understanding of how scenes compare permits the 
computer to learn models for new concepts from 
examples and leads to a new way of thin kina about 

1 earn ing, 

4, These mode is finally endow the machine with the 
ability to recognize instances of previously 

1 earned co ncepts* 

8*3 Background Issues 

In a more cosmic sense, the goat behind this work Is 
to make a machine that can understand the environment just 
as we humans seem to, Some erfties of Artificial 
Intel 1 igencc think that this is not possible, perhaps 
because they cannot Imagine how It can be done, I think 
the real hang up must If* in the understanding one has 
about the notion of u ntf erstanrt Ing* fi review of a few 
dictionaries convinces me that the editors are hard 
pressed to define the word without using it. If fs as if 
it ware a word so basic that it cannot he described in 
& imp 1 er t erms , 

Cut surely to understand must involve the formation 
of a descriptive plat ea u of knowl ed go lying 5 om e w he r e 
between raw, totally unprocessed data and detailed answers 


2 50 


to pro51 ens, I do not wish to belabor this point* but I 
feel that the sort of abstraction represented by the 
network description of a scene can be viewed as 
constituting a sort of understand Ino + If so, depth of 

understanding corresponds roughly to the elaborateness of 
a description* dense networks suggesting more 
understanding than sparse ones. 

Another notion of long standing concern to philosuohy 
Is that of the ideal form. Yet 1 tttle work seems to have 
gone intc careful study of what humans mean by such simple 
concepts as that of the TABLE, I believe study and 
improvement of the concent generator constitutes a fresh 
approach to this prctlen and may lead to interesting new 
re su11 s. 

B, 4 Suggestions for Further Work 

Improvements to and extensions of this work can be 
understood in terms of two extremes: minor change and 
major overhaul. The m inor-c hanne category is largo 
because the highest priority goal in such work must be a 
complete system, A complete system, however f 1 fr-isy, 
serves to guide resource allocation into the nest deserved 
problem areas* Without exger fence with such a system* one 
risks suffering from the phenomenon of diminishing 
returns, expanding great effort for marginal improvements 
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ofl relatively strong pi sees of the system, Put the 
natural result Is that there* isnveh room for further 
Inprov orient of the system's f^rts. Some possibilities 
have already been mentioned, but it fs appropriate to 
nentic-n thorn boro as a convenience for those nho wish to 
i,'0fk in the area* 

1* Nearly all the programs that establish relations 
between objects can be Improved* The' program that 
looks * or support blunders sometimes because 
obvious bottom lines are over looked and s met lines 
because a scene has tipped background objects. 

The program that looks for In -fron t*Of 
relationships cannot handle Situations in which 
objects are aligned, 'iany of these programs could 
benefit from a program that c cl? 1 d imagine hidden 
1 in es + 

2, No distinctions arc made between obvious^ 

unarguable properties and borderline cases. It 
might be good if the analytic pronrars could 
report things like cer ta i n ly-1 eft -of nr sort-of- 
left-of instead of invariable, undifferentiated 
1ef t-of. 

3 „ The rules for the identification of a scene with a 
model need refinenont. The weighting associated 
with the various differences end the way those 
weights are combined have a specious oualltv* It 
imO u Td be fine if so?ne woy c ru Id be devised to 
eliminate the numbers a Itoqether, per ha os thrownh 
a more intelligent procrrain with a built-in 
understanding of priorities. 

4. The schemes for recognizing reasonable clusters of 
objects is particularly primitive and has 
undergone too little testing, Mechanisms must tie 
found for producing and handllnn alternatives to 
the first partition devised, 

6* The entire concept generation procedure and its 
ran if ica tion s certainly should absorb great 


attention, "ore powerful methods fnr recognizing 
important differences are needed. The reach ine 
should have a faculty by which it can cun plain or 
ask questions if the teacher is doing poorly, 
Generalization to functional properties is ^ roust. 

6, The network Hatching program, a It hog oh It is at 
the core of the- entire system* is 3 hastily 
programmed, slow and stubborn StUmbl ebun, Ah 
improved version would s froii! taheously increase the 
power of the many system activities that use it. 

The other kind of improvement is the major overhaul* 
This is not aimed at sophi st lea t fno an existing part of 
the system* hut rather focuses on the spectacular 
Increases in power derived from bolder* bigger ideas* I 
discuss two such possibilities below* Regretg LI y, these 
problems ere difficult to formulate and delimit. 

First is the general problem of introducing bi¬ 
directional information flow. Glance again at the flow 
diagram of figure &-1* Notice that all of the arrows 
point from bottom to tor* indicating that all information 
moves in one direction only. There is as yet no way a 
process can discourse with and modify the behavior of any 
process acting be lew it. 

It seems clear that for hi^h level command to affect 
low level operation In a non-trivial way, there must be 
some alternative or additional! method that can be thrown 
into tattle. As yet few such points of influence exist fn 
my system. There are two left*of right-of procedures, 
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and there Is the option, of using or not using various 
rel a t Ion -f t nd fng and grouping procedures. Eiut the forte 
of an Internally interactive system wi 1! probably involve 
selection of one Method from several pos s 1 bl 1 ft ie s among 
which there are trades -between speed and accuracy. 

Another amorphous problem is that of wedding the 
visual capabilities of this system with other systems that 
specialize in different kinds of perception. A real robot 
should understand the environment not only in terms of 
vision but also In terms of touch, sound* language, and 
perhaps other mediums, Understandinn each of these is a 
major problem, but as work, proceeds* there will be the 
su per ~probl em of understanding how various perceptions of 
the environment should interact to form a unified 
under stand ino „ 

J understand neur oanatom ica 1 tv id once is that 
evolution has tome to or Ins with this problem onl v 
]ately and only in nan with any finesse, Neman 
Geschwfnd reports th&t monkeys have very limited 
ability to correlate things they learn via the 
visual* auditory, and sonesthet fe senses [flj. Indeed 
it may be reasonable to say each monkey is rcallv 
three monkeys occupying the same skull, a visually 
oriented monkey, a auditory one* and a s on esthetic 
one, Man apparently avoids this through a chunk of 
cortex that somehow matches up these perceptions. 

The chunk believed responsible by Geschwird is called 
the inferior parietal lohule. 
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f,pp end ix 

This appendix is a cursory introduction to the 
network mate hi no program essentia 1 to many of t he System's 
operations. Its job is to -determine which nodes of two 
descriptions best correspond. The correspond 1nq- nodes are 
said to be the linked pairs. 

The program starts with the entry nodes of two 
descriptions. ft fmmed iat el y in-nuires if there is 
evidence that the two nodes should bo considered a linked 
pair, 1 he answer is yes if both nodes have a common 
pointer to some common intersection node. Thus X and X 1 
arc a linked pair in figure A -1 „ butt they are certainly 
not in figure A-2, since nodes X and X 1 have neither 
pointers nor nodes in conntn, 

If no 1fnking can occur, the program moves down one 
level through the most common pointer present to dauq liters 
of the currently inspected nodes and tries to find linked 
pairs among them. In figure A-3, the entry nodes are not 
linked on first inspection since there are no 001111*100 
pointers to any common node, They both have daughter 
nodes, however* and these are next examined. P is the 
most numerous pointer* so nodes Cl* C2* CIV, and C2* form 
two groups in which the program tries to find good pairs 
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to establish as lirvkod* {figure A-4) Since Cl and Cl' both 
have the sane pointer to a common node they are good 
candidates for the format Ion of a linked' pa ir , Nodes C 2 
and C£ 1 are even core alike, however, since they have two 
pointers to two common nodes* C on se ouanti y, CZ and CZ ' 
arc linked first, with action on Cl and Cl 1 postponed* 

Each time a pair of nodes is linked,, the program t r v s 
to link up any of the fr daughter nodes that are not 
intersections, In the example this roans that an effort 
is made to find a linked pair between the left node, El* 
and the right nodes, E) f and E£ \ a 11 of which are found 
at the end of P pointers, (figure A - 5} This is the first 
example of a contest, Both El" and 12 1 share temnon 
intersections with Cl, The winning pair picked by the 
machine is always the one with the h In host count of cannon 
pointers to Inter sec ton s or previously linked pairs. In 
this case, El-El 1 scores higher because there Is not only 
a set of pointers to the Intersection node* but also a set 
to the previously linked pair C2-C2 r * 

The linking of El and El" causes examination of their 
daughters, FI on the left, FI r and F2 1 on the right, 
(figure A-6] FI and F2 1 both have a pointer H to a comnon 
node, FI 1 , however, has the must-be version of the 
pointer R to t ho same node* Such satellites occur 
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frequently in norths, indicating mandatary relit ions. 
Priority Is given to matches in which satellites 
correspond to the pointers they are modifications of. 

This means the r with MU ST -EE pointer match between 
nodes FI and FI 1 has somewhat more weight than the match 
of R with R that one wolf Id have between FI and, F 2 " , The 
Fl-Fl' match therefore is considered the better one and 
results in a linked pair* 

FI and FI 1 have no daughters to be paired so no 
further penetration of the net occurs. The next job fs to 
re-exaninc the next higher parent nodes because links just 
formed may provide enough new evidence to link two higher 
nodes. In this case backup first considers nodes El and 
El \ El and El' are already linked. Attention therefore 
pops up still a not her level to Cl and Cl". Cl and C1 " are 
linked, Re-cxamination of their unlinked daughters* 5 and 
E£ ± reveals nothing new. 

Once more the programs attention shifts upward, this 
tine to A and A \ flow there Is A pair of pointers to 
linked nodes supplying linkfng evidence, (figure A-7} A Is 
consequently linked to A * * 

Next comes further examination of the remaining 
daughters of A and At, Q is now the most common pointer 
to unaccounted for nodes, pointing as It does to Cl, fiZ, 
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and S 2 1 . (figure A-a) Among these nodes there Is the same 
amount of evidence for 1 inking HI And C2' as there Is for 
1 i nking G2 and , kfhen this is the case and no further 
evidence can be collected from tliiMna louer level nodes* 
a linking is selected randomly from thnse possible. Here 
G£ and G2 1 are 1 inked, 

This leaves only reexamination of C2 and C.2 1 . The 
intersection evidence is clear ard they are linked. Since 
A and A K have no mere daughters and since they are the 
entry nodes, the process terminates reporting the linkages 
indicated on the fully displayed network of figure* A-9. 
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